Anna_KaRNNa_Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Anna KaRNNa\n",
    "\n",
    "In this notebook, we'll build a character-wise RNN trained on Anna Karenina, one of my all-time favorite books. It'll be able to generate new text based on the text from the book.\n",
    "\n",
    "This network is based off of Andrej Karpathy's [post on RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) and [implementation in Torch](https://github.com/karpathy/char-rnn). Also, some information [here at r2rt](http://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html) and from [Sherjil Ozair](https://github.com/sherjilozair/char-rnn-tensorflow) on GitHub. Below is the general architecture of the character-wise RNN.\n",
    "\n",
    "<img src=\"assets/charseq.jpeg\" width=\"500\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import time\n",
    "from collections import namedtuple\n",
    "\n",
    "import numpy as np\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First we'll load the text file and convert it into integers for our network to use. Here I'm creating a couple dictionaries to convert the characters to and from integers. Encoding the characters as integers makes it easier to use as input in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "with open('don-quijote.txt', 'r') as f:\n",
    "    text=f.read()\n",
    "vocab = sorted(set(text))\n",
    "vocab_to_int = {c: i for i, c in enumerate(vocab)}\n",
    "int_to_vocab = dict(enumerate(vocab))\n",
    "encoded = np.array([vocab_to_int[c] for c in text], dtype=np.int32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Let's check out the first 100 characters, make sure everything is peachy. According to the [American Book Review](http://americanbookreview.org/100bestlines.asp), this is the 6th best first line of a book ever."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\nPrimera parte del ingenioso hidalgo don Quijote de la Mancha\\n\\n\\n\\n\\n\\nCapítulo primero. Que trata de l'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "And we can see the characters encoded as integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  0, 35, 63, 55, 58, 51, 63, 47,  1, 61, 47, 63, 65, 51,  1, 50,\n",
       "       51, 57,  1, 55, 59, 53, 51, 59, 55, 60, 64, 60,  1, 54, 55, 50, 47,\n",
       "       57, 53, 60,  1, 50, 60, 59,  1, 36, 66, 55, 56, 60, 65, 51,  1, 50,\n",
       "       51,  1, 57, 47,  1, 32, 47, 59, 49, 54, 47,  0,  0,  0,  0,  0,  0,\n",
       "       23, 47, 61, 83, 65, 66, 57, 60,  1, 61, 63, 55, 58, 51, 63, 60,  9,\n",
       "        1, 36, 66, 51,  1, 65, 63, 47, 65, 47,  1, 50, 51,  1, 57], dtype=int32)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Since the network is working with individual characters, it's similar to a classification problem in which we are trying to predict the next character from the previous text.  Here's how many 'classes' our network has to pick from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "90"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Making training mini-batches\n",
    "\n",
    "Here is where we'll make our mini-batches for training. Remember that we want our batches to be multiple sequences of some desired number of sequence steps. Considering a simple example, our batches would look like this:\n",
    "\n",
    "<img src=\"assets/sequence_batching@1x.png\" width=500px>\n",
    "\n",
    "\n",
    "<br>\n",
    "We have our text encoded as integers as one long array in `encoded`. Let's create a function that will give us an iterator for our batches. I like using [generator functions](https://jeffknupp.com/blog/2013/04/07/improve-your-python-yield-and-generators-explained/) to do this. Then we can pass `encoded` into this function and get our batch generator.\n",
    "\n",
    "The first thing we need to do is discard some of the text so we only have completely full batches. Each batch contains $N \\times M$ characters, where $N$ is the batch size (the number of sequences) and $M$ is the number of steps. Then, to get the number of batches we can make from some array `arr`, you divide the length of `arr` by the batch size. Once you know the number of batches and the batch size, you can get the total number of characters to keep.\n",
    "\n",
    "After that, we need to split `arr` into $N$ sequences. You can do this using `arr.reshape(size)` where `size` is a tuple containing the dimensions sizes of the reshaped array. We know we want $N$ sequences (`n_seqs` below), let's make that the size of the first dimension. For the second dimension, you can use `-1` as a placeholder in the size, it'll fill up the array with the appropriate data for you. After this, you should have an array that is $N \\times (M * K)$ where $K$ is the number of batches.\n",
    "\n",
    "Now that we have this array, we can iterate through it to get our batches. The idea is each batch is a $N \\times M$ window on the array. For each subsequent batch, the window moves over by `n_steps`. We also want to create both the input and target arrays. Remember that the targets are the inputs shifted over one character. You'll usually see the first input character used as the last target character, so something like this:\n",
    "```python\n",
    "y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]\n",
    "```\n",
    "where `x` is the input batch and `y` is the target batch.\n",
    "\n",
    "The way I like to do this window is use `range` to take steps of size `n_steps` from $0$ to `arr.shape[1]`, the total number of steps in each sequence. That way, the integers you get from `range` always point to the start of a batch, and each window is `n_steps` wide.\n",
    "\n",
    "> **Exercise:** Write the code for creating batches in the function below. The exercises in this notebook _will not be easy_. I've provided a notebook with solutions alongside this notebook. If you get stuck, checkout the solutions. The most important thing is that you don't copy and paste the code into here, **type out the solution code yourself.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_batches(arr, n_seqs, n_steps):\n",
    "    '''Create a generator that returns batches of size\n",
    "       n_seqs x n_steps from arr.\n",
    "       \n",
    "       Arguments\n",
    "       ---------\n",
    "       arr: Array you want to make batches from\n",
    "       n_seqs: Batch size, the number of sequences per batch\n",
    "       n_steps: Number of sequence steps per batch\n",
    "    '''\n",
    "    # Get the number of characters per batch and number of batches we can make\n",
    "    characters_per_batch = n_seqs * n_steps\n",
    "    n_batches = len(arr)//characters_per_batch\n",
    "    \n",
    "    # Keep only enough characters to make full batches\n",
    "    arr = arr[:characters_per_batch * n_batches]\n",
    "    \n",
    "    # Reshape into n_seqs rows\n",
    "    arr = arr.reshape((n_seqs, -1))\n",
    "    \n",
    "    for n in range(0, arr.shape[1], n_steps):\n",
    "        # The features\n",
    "        x = arr[:, n: n+n_steps]\n",
    "        # The targets, shifted by one\n",
    "        y = np.zeros_like(x)\n",
    "        y[:, :-1], y[:, -1] = x[:, 1:], x[:, 0]\n",
    "        yield x, y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now I'll make my data sets and we can check out what's going on here. Here I'm going to use a batch size of 10 and 50 sequence steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "batches = get_batches(encoded, 10, 50)\n",
    "x, y = next(batches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x\n",
      " [[ 0  0 35 63 55 58 51 63 47  1]\n",
      " [ 1 65 82 63 58 55 59 60 64  7]\n",
      " [58 60  1 62 66 51 50 47 48 47]\n",
      " [47 48 83 47  1 64 55 50 60  1]\n",
      " [51 50 55 63 58 51  1 50 82 57]\n",
      " [55 51 63 60 59  1 61 60 63  1]\n",
      " [66 51  1 49 60 59  0 51 64 65]\n",
      " [ 1 50 51  1 64 51 53 66 55 63]\n",
      " [50 47 59 65 51  7  1 55 59 49]\n",
      " [50 51 56 47 59 50 60  1 47  1]]\n",
      "\n",
      "y\n",
      " [[ 0 35 63 55 58 51 63 47  1 61]\n",
      " [65 82 63 58 55 59 60 64  7  1]\n",
      " [60  1 62 66 51 50 47 48 47  1]\n",
      " [48 83 47  1 64 55 50 60  1 55]\n",
      " [50 55 63 58 51  1 50 82 57  7]\n",
      " [51 63 60 59  1 61 60 63  1 64]\n",
      " [51  1 49 60 59  0 51 64 65 47]\n",
      " [50 51  1 64 51 53 66 55 63 57]\n",
      " [47 59 65 51  7  1 55 59 49 57]\n",
      " [51 56 47 59 50 60  1 47  1 64]]\n"
     ]
    }
   ],
   "source": [
    "print('x\\n', x[:10, :10])\n",
    "print('\\ny\\n', y[:10, :10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you implemented `get_batches` correctly, the above output should look something like \n",
    "```\n",
    "x\n",
    " [[55 63 69 22  6 76 45  5 16 35]\n",
    " [ 5 69  1  5 12 52  6  5 56 52]\n",
    " [48 29 12 61 35 35  8 64 76 78]\n",
    " [12  5 24 39 45 29 12 56  5 63]\n",
    " [ 5 29  6  5 29 78 28  5 78 29]\n",
    " [ 5 13  6  5 36 69 78 35 52 12]\n",
    " [63 76 12  5 18 52  1 76  5 58]\n",
    " [34  5 73 39  6  5 12 52 36  5]\n",
    " [ 6  5 29 78 12 79  6 61  5 59]\n",
    " [ 5 78 69 29 24  5  6 52  5 63]]\n",
    "\n",
    "y\n",
    " [[63 69 22  6 76 45  5 16 35 35]\n",
    " [69  1  5 12 52  6  5 56 52 29]\n",
    " [29 12 61 35 35  8 64 76 78 28]\n",
    " [ 5 24 39 45 29 12 56  5 63 29]\n",
    " [29  6  5 29 78 28  5 78 29 45]\n",
    " [13  6  5 36 69 78 35 52 12 43]\n",
    " [76 12  5 18 52  1 76  5 58 52]\n",
    " [ 5 73 39  6  5 12 52 36  5 78]\n",
    " [ 5 29 78 12 79  6 61  5 59 63]\n",
    " [78 69 29 24  5  6 52  5 63 76]]\n",
    " ```\n",
    " although the exact numbers will be different. Check to make sure the data is shifted over one step for `y`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Building the model\n",
    "\n",
    "Below is where you'll build the network. We'll break it up into parts so it's easier to reason about each bit. Then we can connect them up into the whole network.\n",
    "\n",
    "<img src=\"assets/charRNN.png\" width=500px>\n",
    "\n",
    "\n",
    "### Inputs\n",
    "\n",
    "First off we'll create our input placeholders. As usual we need placeholders for the training data and the targets. We'll also create a placeholder for dropout layers called `keep_prob`. This will be a scalar, that is a 0-D tensor. To make a scalar, you create a placeholder without giving it a size.\n",
    "\n",
    "> **Exercise:** Create the input placeholders in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def build_inputs(batch_size, num_steps):\n",
    "    ''' Define placeholders for inputs, targets, and dropout \n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        batch_size: Batch size, number of sequences per batch\n",
    "        num_steps: Number of sequence steps in a batch\n",
    "        \n",
    "    '''\n",
    "    # Declare placeholders we'll feed into the graph\n",
    "    inputs = tf.placeholder(tf.int32, shape=(batch_size, num_steps), name=\"input\")\n",
    "    targets = tf.placeholder(tf.int32, shape=(batch_size, num_steps), name=\"targets\")\n",
    "    \n",
    "    # Keep probability placeholder for drop out layers\n",
    "    keep_prob = tf.placeholder(tf.float32, name=\"keep_prob\")\n",
    "    \n",
    "    return inputs, targets, keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### LSTM Cell\n",
    "\n",
    "Here we will create the LSTM cell we'll use in the hidden layer. We'll use this cell as a building block for the RNN. So we aren't actually defining the RNN here, just the type of cell we'll use in the hidden layer.\n",
    "\n",
    "We first create a basic LSTM cell with\n",
    "\n",
    "```python\n",
    "lstm = tf.contrib.rnn.BasicLSTMCell(num_units)\n",
    "```\n",
    "\n",
    "where `num_units` is the number of units in the hidden layers in the cell. Then we can add dropout by wrapping it with \n",
    "\n",
    "```python\n",
    "tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "```\n",
    "You pass in a cell and it will automatically add dropout to the inputs or outputs. Finally, we can stack up the LSTM cells into layers with [`tf.contrib.rnn.MultiRNNCell`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/contrib/rnn/MultiRNNCell). With this, you pass in a list of cells and it will send the output of one cell into the next cell. Previously with TensorFlow 1.0, you could do this\n",
    "\n",
    "```python\n",
    "tf.contrib.rnn.MultiRNNCell([cell]*num_layers)\n",
    "```\n",
    "\n",
    "This might look a little weird if you know Python well because this will create a list of the same `cell` object. However, TensorFlow 1.0 will create different weight matrices for all `cell` objects. But, starting with TensorFlow 1.1 you actually need to create new cell objects in the list. To get it to work in TensorFlow 1.1, it should look like\n",
    "\n",
    "```python\n",
    "def build_cell(num_units, keep_prob):\n",
    "    lstm = tf.contrib.rnn.BasicLSTMCell(num_units)\n",
    "    drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "    \n",
    "    return drop\n",
    "    \n",
    "tf.contrib.rnn.MultiRNNCell([build_cell(num_units, keep_prob) for _ in range(num_layers)])\n",
    "```\n",
    "\n",
    "Even though this is actually multiple LSTM cells stacked on each other, you can treat the multiple layers as one cell.\n",
    "\n",
    "We also need to create an initial cell state of all zeros. This can be done like so\n",
    "\n",
    "```python\n",
    "initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "```\n",
    "\n",
    "Below, we implement the `build_lstm` function to create these LSTM cells and the initial state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def build_lstm(lstm_size, num_layers, batch_size, keep_prob):\n",
    "    ''' Build LSTM cell.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        keep_prob: Scalar tensor (tf.placeholder) for the dropout keep probability\n",
    "        lstm_size: Size of the hidden layers in the LSTM cells\n",
    "        num_layers: Number of LSTM layers\n",
    "        batch_size: Batch size\n",
    "\n",
    "    '''\n",
    "    ### Build the LSTM Cell\n",
    "    # Use a basic LSTM cell\n",
    "    # lstm = \n",
    "    \n",
    "    # Add dropout to the cell outputs\n",
    "    # drop = \n",
    "    \n",
    "    # Stack up multiple LSTM layers, for deep learning\n",
    "\n",
    "    def build_cell(num_units, keep_prob):\n",
    "        lstm = tf.contrib.rnn.BasicLSTMCell(lstm_size)\n",
    "        drop = tf.contrib.rnn.DropoutWrapper(lstm, output_keep_prob=keep_prob)\n",
    "\n",
    "        return drop\n",
    "\n",
    "    cell = tf.contrib.rnn.MultiRNNCell([build_cell(lstm_size, keep_prob) for _ in range(num_layers)])\n",
    "    initial_state = cell.zero_state(batch_size, tf.float32)\n",
    "    \n",
    "    return cell, initial_state"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### RNN Output\n",
    "\n",
    "Here we'll create the output layer. We need to connect the output of the RNN cells to a full connected layer with a softmax output. The softmax output gives us a probability distribution we can use to predict the next character, so we want this layer to have size $C$, the number of classes/characters we have in our text.\n",
    "\n",
    "If our input has batch size $N$, number of steps $M$, and the hidden layer has $L$ hidden units, then the output is a 3D tensor with size $N \\times M \\times L$. The output of each LSTM cell has size $L$, we have $M$ of them, one for each sequence step, and we have $N$ sequences. So the total size is $N \\times M \\times L$. \n",
    "\n",
    "We are using the same fully connected layer, the same weights, for each of the outputs. Then, to make things easier, we should reshape the outputs into a 2D tensor with shape $(M * N) \\times L$. That is, one row for each sequence and step, where the values of each row are the output from the LSTM cells. We get the LSTM output as a list, `lstm_output`. First we need to concatenate this whole list into one array with [`tf.concat`](https://www.tensorflow.org/api_docs/python/tf/concat). Then, reshape it (with `tf.reshape`) to size $(M * N) \\times L$.\n",
    "\n",
    "One we have the outputs reshaped, we can do the matrix multiplication with the weights. We need to wrap the weight and bias variables in a variable scope with `tf.variable_scope(scope_name)` because there are weights being created in the LSTM cells. TensorFlow will throw an error if the weights created here have the same names as the weights created in the LSTM cells, which they will be default. To avoid this, we wrap the variables in a variable scope so we can give them unique names.\n",
    "\n",
    "> **Exercise:** Implement the output layer in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def build_output(lstm_output, in_size, out_size):\n",
    "    ''' Build a softmax layer, return the softmax output and logits.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        \n",
    "        lstm_output: List of output tensors from the LSTM layer\n",
    "        in_size: Size of the input tensor, for example, size of the LSTM cells\n",
    "        out_size: Size of this softmax layer\n",
    "    \n",
    "    '''\n",
    "\n",
    "    # Reshape output so it's a bunch of rows, one row for each step for each sequence.\n",
    "    # Concatenate lstm_output over axis 1 (the columns)\n",
    "    seq_output = tf.concat(lstm_output, axis=1)\n",
    "    # Reshape seq_output to a 2D tensor with lstm_size columns\n",
    "    x = tf.reshape(seq_output, [-1, in_size])\n",
    "    \n",
    "    # Connect the RNN outputs to a softmax layer\n",
    "    with tf.variable_scope('softmax'):\n",
    "        # Create the weight and bias variables here\n",
    "        softmax_w = tf.Variable(tf.truncated_normal([in_size, out_size], stddev=0.1))\n",
    "        softmax_b = tf.Variable(tf.zeros([out_size]))\n",
    "    \n",
    "    # Since output is a bunch of rows of RNN cell outputs, logits will be a bunch\n",
    "    # of rows of logit outputs, one for each step and sequence\n",
    "    logits = tf.add(tf.matmul(x, softmax_w), softmax_b)\n",
    "    \n",
    "    # Use softmax to get the probabilities for predicted characters\n",
    "    out = tf.nn.softmax(logits, name=\"prediction\")\n",
    "    \n",
    "    return out, logits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Training loss\n",
    "\n",
    "Next up is the training loss. We get the logits and targets and calculate the softmax cross-entropy loss. First we need to one-hot encode the targets, we're getting them as encoded characters. Then, reshape the one-hot targets so it's a 2D tensor with size $(M*N) \\times C$ where $C$ is the number of classes/characters we have. Remember that we reshaped the LSTM outputs and ran them through a fully connected layer with $C$ units. So our logits will also have size $(M*N) \\times C$.\n",
    "\n",
    "Then we run the logits and targets through `tf.nn.softmax_cross_entropy_with_logits` and find the mean to get the loss.\n",
    "\n",
    ">**Exercise:** Implement the loss calculation in the function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def build_loss(logits, targets, lstm_size, num_classes):\n",
    "    ''' Calculate the loss from the logits and the targets.\n",
    "    \n",
    "        Arguments\n",
    "        ---------\n",
    "        logits: Logits from final fully connected layer\n",
    "        targets: Targets for supervised learning\n",
    "        lstm_size: Number of LSTM hidden units\n",
    "        num_classes: Number of classes in targets\n",
    "        \n",
    "    '''\n",
    "    \n",
    "    # One-hot encode targets and reshape to match logits, one row per sequence per step\n",
    "    y_one_hot = tf.one_hot(targets, num_classes)\n",
    "    y_reshaped = tf.reshape(y_one_hot, logits.get_shape())\n",
    "    \n",
    "    # Softmax cross entropy loss\n",
    "    loss = tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=y_reshaped)\n",
    "    loss = tf.reduce_mean(loss)\n",
    "    \n",
    "    return loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Optimizer\n",
    "\n",
    "Here we build the optimizer. Normal RNNs have have issues gradients exploding and disappearing. LSTMs fix the disappearance problem, but the gradients can still grow without bound. To fix this, we can clip the gradients above some threshold. That is, if a gradient is larger than that threshold, we set it to the threshold. This will ensure the gradients never grow overly large. Then we use an AdamOptimizer for the learning step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def build_optimizer(loss, learning_rate, grad_clip):\n",
    "    ''' Build optmizer for training, using gradient clipping.\n",
    "    \n",
    "        Arguments:\n",
    "        loss: Network loss\n",
    "        learning_rate: Learning rate for optimizer\n",
    "    \n",
    "    '''\n",
    "    \n",
    "    # Optimizer for training, using gradient clipping to control exploding gradients\n",
    "    tvars = tf.trainable_variables()\n",
    "    grads, _ = tf.clip_by_global_norm(tf.gradients(loss, tvars), grad_clip)\n",
    "    train_op = tf.train.AdamOptimizer(learning_rate)\n",
    "    optimizer = train_op.apply_gradients(zip(grads, tvars))\n",
    "    \n",
    "    return optimizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Build the network\n",
    "\n",
    "Now we can put all the pieces together and build a class for the network. To actually run data through the LSTM cells, we will use [`tf.nn.dynamic_rnn`](https://www.tensorflow.org/versions/r1.0/api_docs/python/tf/nn/dynamic_rnn). This function will pass the hidden and cell states across LSTM cells appropriately for us. It returns the outputs for each LSTM cell at each step for each sequence in the mini-batch. It also gives us the final LSTM state. We want to save this state as `final_state` so we can pass it to the first LSTM cell in the the next mini-batch run. For `tf.nn.dynamic_rnn`, we pass in the cell and initial state we get from `build_lstm`, as well as our input sequences. Also, we need to one-hot encode the inputs before going into the RNN. \n",
    "\n",
    "> **Exercise:** Use the functions you've implemented previously and `tf.nn.dynamic_rnn` to build the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class CharRNN:\n",
    "    \n",
    "    def __init__(self, num_classes, batch_size=64, num_steps=50, \n",
    "                       lstm_size=128, num_layers=2, learning_rate=0.001, \n",
    "                       grad_clip=5, sampling=False):\n",
    "    \n",
    "        # When we're using this network for sampling later, we'll be passing in\n",
    "        # one character at a time, so providing an option for that\n",
    "        if sampling == True:\n",
    "            batch_size, num_steps = 1, 1\n",
    "        else:\n",
    "            batch_size, num_steps = batch_size, num_steps\n",
    "\n",
    "        tf.reset_default_graph()\n",
    "        \n",
    "        # Build the input placeholder tensors\n",
    "        self.inputs, self.targets, self.keep_prob = build_inputs(batch_size, num_steps)\n",
    "\n",
    "        # Build the LSTM cell\n",
    "        cell, self.initial_state = build_lstm(lstm_size, num_layers, batch_size, self.keep_prob)\n",
    "\n",
    "        ### Run the data through the RNN layers\n",
    "        # First, one-hot encode the input tokens\n",
    "        x_one_hot = tf.one_hot(self.inputs, num_classes)\n",
    "        \n",
    "        # Run each sequence step through the RNN with tf.nn.dynamic_rnn \n",
    "        outputs, state = tf.nn.dynamic_rnn(cell, x_one_hot, initial_state=self.initial_state)\n",
    "        self.final_state = state\n",
    "        \n",
    "        # Get softmax predictions and logits\n",
    "        self.prediction, self.logits = build_output(outputs, lstm_size, num_classes)\n",
    "        \n",
    "        # Loss and optimizer (with gradient clipping)\n",
    "        self.loss = build_loss(self.logits, self.targets, lstm_size, num_classes)\n",
    "        self.optimizer = build_optimizer(self.loss, learning_rate, grad_clip)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Hyperparameters\n",
    "\n",
    "Here are the hyperparameters for the network.\n",
    "\n",
    "* `batch_size` - Number of sequences running through the network in one pass.\n",
    "* `num_steps` - Number of characters in the sequence the network is trained on. Larger is better typically, the network will learn more long range dependencies. But it takes longer to train. 100 is typically a good number here.\n",
    "* `lstm_size` - The number of units in the hidden layers.\n",
    "* `num_layers` - Number of hidden LSTM layers to use\n",
    "* `learning_rate` - Learning rate for training\n",
    "* `keep_prob` - The dropout keep probability when training. If you're network is overfitting, try decreasing this.\n",
    "\n",
    "Here's some good advice from Andrej Karpathy on training the network. I'm going to copy it in here for your benefit, but also link to [where it originally came from](https://github.com/karpathy/char-rnn#tips-and-tricks).\n",
    "\n",
    "> ## Tips and Tricks\n",
    "\n",
    ">### Monitoring Validation Loss vs. Training Loss\n",
    ">If you're somewhat new to Machine Learning or Neural Networks it can take a bit of expertise to get good models. The most important quantity to keep track of is the difference between your training loss (printed during training) and the validation loss (printed once in a while when the RNN is run on the validation data (by default every 1000 iterations)). In particular:\n",
    "\n",
    "> - If your training loss is much lower than validation loss then this means the network might be **overfitting**. Solutions to this are to decrease your network size, or to increase dropout. For example you could try dropout of 0.5 and so on.\n",
    "> - If your training/validation loss are about equal then your model is **underfitting**. Increase the size of your model (either number of layers or the raw number of neurons per layer)\n",
    "\n",
    "> ### Approximate number of parameters\n",
    "\n",
    "> The two most important parameters that control the model are `lstm_size` and `num_layers`. I would advise that you always use `num_layers` of either 2/3. The `lstm_size` can be adjusted based on how much data you have. The two important quantities to keep track of here are:\n",
    "\n",
    "> - The number of parameters in your model. This is printed when you start training.\n",
    "> - The size of your dataset. 1MB file is approximately 1 million characters.\n",
    "\n",
    ">These two should be about the same order of magnitude. It's a little tricky to tell. Here are some examples:\n",
    "\n",
    "> - I have a 100MB dataset and I'm using the default parameter settings (which currently print 150K parameters). My data size is significantly larger (100 mil >> 0.15 mil), so I expect to heavily underfit. I am thinking I can comfortably afford to make `lstm_size` larger.\n",
    "> - I have a 10MB dataset and running a 10 million parameter model. I'm slightly nervous and I'm carefully monitoring my validation loss. If it's larger than my training loss then I may want to try to increase dropout a bit and see if that helps the validation loss.\n",
    "\n",
    "> ### Best models strategy\n",
    "\n",
    ">The winning strategy to obtaining very good models (if you have the compute time) is to always err on making the network larger (as large as you're willing to wait for it to compute) and then try different dropout values (between 0,1). Whatever model has the best validation performance (the loss, written in the checkpoint filename, low is good) is the one you should use in the end.\n",
    "\n",
    ">It is very common in deep learning to run many different models with many different hyperparameter settings, and in the end take whatever checkpoint gave the best validation performance.\n",
    "\n",
    ">By the way, the size of your training and validation splits are also parameters. Make sure you have a decent amount of data in your validation set or otherwise the validation performance will be noisy and not very informative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "batch_size = 100        # Sequences per batch\n",
    "num_steps = 100         # Number of sequence steps per batch\n",
    "lstm_size = 512         # Size of hidden layers in LSTMs\n",
    "num_layers = 2          # Number of LSTM layers\n",
    "learning_rate = 0.001   # Learning rate\n",
    "keep_prob = 0.5         # Dropout keep probability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Time for training\n",
    "\n",
    "This is typical training code, passing inputs and targets into the network, then running the optimizer. Here we also get back the final LSTM state for the mini-batch. Then, we pass that state back into the network so the next batch can continue the state from the previous batch. And every so often (set by `save_every_n`) I save a checkpoint.\n",
    "\n",
    "Here I'm saving checkpoints with the format\n",
    "\n",
    "`i{iteration number}_l{# hidden layer units}.ckpt`\n",
    "\n",
    "> **Exercise:** Set the hyperparameters above to train the network. Watch the training loss, it should be consistently dropping. Also, I highly advise running this on a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 1/20...  Training Step: 1...  Training loss: 4.4991...  6.3301 sec/batch\n",
      "Epoch: 1/20...  Training Step: 2...  Training loss: 4.3982...  6.2324 sec/batch\n",
      "Epoch: 1/20...  Training Step: 3...  Training loss: 3.8244...  6.2283 sec/batch\n",
      "Epoch: 1/20...  Training Step: 4...  Training loss: 5.0819...  6.2664 sec/batch\n",
      "Epoch: 1/20...  Training Step: 5...  Training loss: 4.3866...  6.2201 sec/batch\n",
      "Epoch: 1/20...  Training Step: 6...  Training loss: 3.9643...  6.2197 sec/batch\n",
      "Epoch: 1/20...  Training Step: 7...  Training loss: 3.6788...  6.2932 sec/batch\n",
      "Epoch: 1/20...  Training Step: 8...  Training loss: 3.5896...  6.2119 sec/batch\n",
      "Epoch: 1/20...  Training Step: 9...  Training loss: 3.4702...  6.2434 sec/batch\n",
      "Epoch: 1/20...  Training Step: 10...  Training loss: 3.4087...  6.2237 sec/batch\n",
      "Epoch: 1/20...  Training Step: 11...  Training loss: 3.3571...  6.2172 sec/batch\n",
      "Epoch: 1/20...  Training Step: 12...  Training loss: 3.3022...  6.2380 sec/batch\n",
      "Epoch: 1/20...  Training Step: 13...  Training loss: 3.3137...  6.2671 sec/batch\n",
      "Epoch: 1/20...  Training Step: 14...  Training loss: 3.2962...  6.2618 sec/batch\n",
      "Epoch: 1/20...  Training Step: 15...  Training loss: 3.2775...  6.3107 sec/batch\n",
      "Epoch: 1/20...  Training Step: 16...  Training loss: 3.2798...  6.2962 sec/batch\n",
      "Epoch: 1/20...  Training Step: 17...  Training loss: 3.2535...  6.1604 sec/batch\n",
      "Epoch: 1/20...  Training Step: 18...  Training loss: 3.2311...  6.2992 sec/batch\n",
      "Epoch: 1/20...  Training Step: 19...  Training loss: 3.2545...  6.2343 sec/batch\n",
      "Epoch: 1/20...  Training Step: 20...  Training loss: 3.2266...  6.2283 sec/batch\n",
      "Epoch: 1/20...  Training Step: 21...  Training loss: 3.2052...  6.2406 sec/batch\n",
      "Epoch: 1/20...  Training Step: 22...  Training loss: 3.2171...  6.2569 sec/batch\n",
      "Epoch: 1/20...  Training Step: 23...  Training loss: 3.1846...  6.1939 sec/batch\n",
      "Epoch: 1/20...  Training Step: 24...  Training loss: 3.2070...  6.2309 sec/batch\n",
      "Epoch: 1/20...  Training Step: 25...  Training loss: 3.1856...  6.2224 sec/batch\n",
      "Epoch: 1/20...  Training Step: 26...  Training loss: 3.1845...  6.2942 sec/batch\n",
      "Epoch: 1/20...  Training Step: 27...  Training loss: 3.1657...  6.2704 sec/batch\n",
      "Epoch: 1/20...  Training Step: 28...  Training loss: 3.1395...  6.2514 sec/batch\n",
      "Epoch: 1/20...  Training Step: 29...  Training loss: 3.1541...  6.2768 sec/batch\n",
      "Epoch: 1/20...  Training Step: 30...  Training loss: 3.1443...  6.3135 sec/batch\n",
      "Epoch: 1/20...  Training Step: 31...  Training loss: 3.1317...  6.1965 sec/batch\n",
      "Epoch: 1/20...  Training Step: 32...  Training loss: 3.1528...  6.1799 sec/batch\n",
      "Epoch: 1/20...  Training Step: 33...  Training loss: 3.1402...  6.2852 sec/batch\n",
      "Epoch: 1/20...  Training Step: 34...  Training loss: 3.1486...  6.2634 sec/batch\n",
      "Epoch: 1/20...  Training Step: 35...  Training loss: 3.1439...  6.2703 sec/batch\n",
      "Epoch: 1/20...  Training Step: 36...  Training loss: 3.1033...  6.2251 sec/batch\n",
      "Epoch: 1/20...  Training Step: 37...  Training loss: 3.1160...  6.2344 sec/batch\n",
      "Epoch: 1/20...  Training Step: 38...  Training loss: 3.1208...  6.2645 sec/batch\n",
      "Epoch: 1/20...  Training Step: 39...  Training loss: 3.1410...  6.2075 sec/batch\n",
      "Epoch: 1/20...  Training Step: 40...  Training loss: 3.1200...  6.2309 sec/batch\n",
      "Epoch: 1/20...  Training Step: 41...  Training loss: 3.1449...  6.2189 sec/batch\n",
      "Epoch: 1/20...  Training Step: 42...  Training loss: 3.1066...  6.3020 sec/batch\n",
      "Epoch: 1/20...  Training Step: 43...  Training loss: 3.1183...  6.3134 sec/batch\n",
      "Epoch: 1/20...  Training Step: 44...  Training loss: 3.1491...  6.3017 sec/batch\n",
      "Epoch: 1/20...  Training Step: 45...  Training loss: 3.1241...  6.2608 sec/batch\n",
      "Epoch: 1/20...  Training Step: 46...  Training loss: 3.1111...  6.2112 sec/batch\n",
      "Epoch: 1/20...  Training Step: 47...  Training loss: 3.0892...  6.2554 sec/batch\n",
      "Epoch: 1/20...  Training Step: 48...  Training loss: 3.1079...  6.2476 sec/batch\n",
      "Epoch: 1/20...  Training Step: 49...  Training loss: 3.1145...  6.2049 sec/batch\n",
      "Epoch: 1/20...  Training Step: 50...  Training loss: 3.1044...  6.1868 sec/batch\n",
      "Epoch: 1/20...  Training Step: 51...  Training loss: 3.1097...  6.2760 sec/batch\n",
      "Epoch: 1/20...  Training Step: 52...  Training loss: 3.1001...  6.2329 sec/batch\n",
      "Epoch: 1/20...  Training Step: 53...  Training loss: 3.0949...  6.2524 sec/batch\n",
      "Epoch: 1/20...  Training Step: 54...  Training loss: 3.0916...  6.2657 sec/batch\n",
      "Epoch: 1/20...  Training Step: 55...  Training loss: 3.0998...  6.2845 sec/batch\n",
      "Epoch: 1/20...  Training Step: 56...  Training loss: 3.0939...  6.2822 sec/batch\n",
      "Epoch: 1/20...  Training Step: 57...  Training loss: 3.0542...  6.2772 sec/batch\n",
      "Epoch: 1/20...  Training Step: 58...  Training loss: 3.1021...  6.2263 sec/batch\n",
      "Epoch: 1/20...  Training Step: 59...  Training loss: 3.0916...  6.2589 sec/batch\n",
      "Epoch: 1/20...  Training Step: 60...  Training loss: 3.1002...  6.2568 sec/batch\n",
      "Epoch: 1/20...  Training Step: 61...  Training loss: 3.0777...  6.2303 sec/batch\n",
      "Epoch: 1/20...  Training Step: 62...  Training loss: 3.0939...  6.2257 sec/batch\n",
      "Epoch: 1/20...  Training Step: 63...  Training loss: 3.0873...  6.2326 sec/batch\n",
      "Epoch: 1/20...  Training Step: 64...  Training loss: 3.0658...  6.1863 sec/batch\n",
      "Epoch: 1/20...  Training Step: 65...  Training loss: 3.0678...  6.1953 sec/batch\n",
      "Epoch: 1/20...  Training Step: 66...  Training loss: 3.0880...  6.2507 sec/batch\n",
      "Epoch: 1/20...  Training Step: 67...  Training loss: 3.0766...  6.2594 sec/batch\n",
      "Epoch: 1/20...  Training Step: 68...  Training loss: 3.0897...  6.2319 sec/batch\n",
      "Epoch: 1/20...  Training Step: 69...  Training loss: 3.1023...  6.2852 sec/batch\n",
      "Epoch: 1/20...  Training Step: 70...  Training loss: 3.0820...  6.3156 sec/batch\n",
      "Epoch: 1/20...  Training Step: 71...  Training loss: 3.0741...  6.2741 sec/batch\n",
      "Epoch: 1/20...  Training Step: 72...  Training loss: 3.0920...  6.2636 sec/batch\n",
      "Epoch: 1/20...  Training Step: 73...  Training loss: 3.0718...  6.2710 sec/batch\n",
      "Epoch: 1/20...  Training Step: 74...  Training loss: 3.0765...  6.1979 sec/batch\n",
      "Epoch: 1/20...  Training Step: 75...  Training loss: 3.0627...  6.2287 sec/batch\n",
      "Epoch: 1/20...  Training Step: 76...  Training loss: 3.0533...  6.2442 sec/batch\n",
      "Epoch: 1/20...  Training Step: 77...  Training loss: 3.0683...  6.2029 sec/batch\n",
      "Epoch: 1/20...  Training Step: 78...  Training loss: 3.0651...  6.2213 sec/batch\n",
      "Epoch: 1/20...  Training Step: 79...  Training loss: 3.0690...  6.2124 sec/batch\n",
      "Epoch: 1/20...  Training Step: 80...  Training loss: 3.0566...  6.2246 sec/batch\n",
      "Epoch: 1/20...  Training Step: 81...  Training loss: 3.0511...  6.1786 sec/batch\n",
      "Epoch: 1/20...  Training Step: 82...  Training loss: 3.0528...  6.2573 sec/batch\n",
      "Epoch: 1/20...  Training Step: 83...  Training loss: 3.0542...  6.2290 sec/batch\n",
      "Epoch: 1/20...  Training Step: 84...  Training loss: 3.0576...  6.2063 sec/batch\n",
      "Epoch: 1/20...  Training Step: 85...  Training loss: 3.0559...  6.3137 sec/batch\n",
      "Epoch: 1/20...  Training Step: 86...  Training loss: 3.0563...  6.2462 sec/batch\n",
      "Epoch: 1/20...  Training Step: 87...  Training loss: 3.0281...  6.2240 sec/batch\n",
      "Epoch: 1/20...  Training Step: 88...  Training loss: 3.0546...  6.2666 sec/batch\n",
      "Epoch: 1/20...  Training Step: 89...  Training loss: 3.0434...  6.2131 sec/batch\n",
      "Epoch: 1/20...  Training Step: 90...  Training loss: 3.0530...  6.2350 sec/batch\n",
      "Epoch: 1/20...  Training Step: 91...  Training loss: 3.0352...  6.1918 sec/batch\n",
      "Epoch: 1/20...  Training Step: 92...  Training loss: 3.0734...  6.2490 sec/batch\n",
      "Epoch: 1/20...  Training Step: 93...  Training loss: 3.0307...  6.2095 sec/batch\n",
      "Epoch: 1/20...  Training Step: 94...  Training loss: 3.0814...  6.2605 sec/batch\n",
      "Epoch: 1/20...  Training Step: 95...  Training loss: 3.0448...  6.2684 sec/batch\n",
      "Epoch: 1/20...  Training Step: 96...  Training loss: 3.0706...  6.2449 sec/batch\n",
      "Epoch: 1/20...  Training Step: 97...  Training loss: 3.0540...  6.2455 sec/batch\n",
      "Epoch: 1/20...  Training Step: 98...  Training loss: 3.0387...  6.2295 sec/batch\n",
      "Epoch: 1/20...  Training Step: 99...  Training loss: 3.0469...  6.2339 sec/batch\n",
      "Epoch: 1/20...  Training Step: 100...  Training loss: 3.0563...  6.2833 sec/batch\n",
      "Epoch: 1/20...  Training Step: 101...  Training loss: 3.0402...  6.2290 sec/batch\n",
      "Epoch: 1/20...  Training Step: 102...  Training loss: 3.0298...  6.2682 sec/batch\n",
      "Epoch: 1/20...  Training Step: 103...  Training loss: 3.0390...  6.2663 sec/batch\n",
      "Epoch: 1/20...  Training Step: 104...  Training loss: 3.0576...  6.2491 sec/batch\n",
      "Epoch: 1/20...  Training Step: 105...  Training loss: 3.0301...  6.2142 sec/batch\n",
      "Epoch: 1/20...  Training Step: 106...  Training loss: 3.0228...  6.2536 sec/batch\n",
      "Epoch: 1/20...  Training Step: 107...  Training loss: 3.0307...  6.2269 sec/batch\n",
      "Epoch: 1/20...  Training Step: 108...  Training loss: 3.0372...  6.2063 sec/batch\n",
      "Epoch: 1/20...  Training Step: 109...  Training loss: 3.0040...  6.1719 sec/batch\n",
      "Epoch: 1/20...  Training Step: 110...  Training loss: 3.0023...  6.2498 sec/batch\n",
      "Epoch: 1/20...  Training Step: 111...  Training loss: 3.0080...  6.2270 sec/batch\n",
      "Epoch: 1/20...  Training Step: 112...  Training loss: 3.0277...  6.2081 sec/batch\n",
      "Epoch: 1/20...  Training Step: 113...  Training loss: 3.0132...  6.2691 sec/batch\n",
      "Epoch: 1/20...  Training Step: 114...  Training loss: 2.9961...  6.2618 sec/batch\n",
      "Epoch: 1/20...  Training Step: 115...  Training loss: 2.9975...  6.2491 sec/batch\n",
      "Epoch: 1/20...  Training Step: 116...  Training loss: 2.9867...  6.2376 sec/batch\n",
      "Epoch: 1/20...  Training Step: 117...  Training loss: 3.0077...  6.2875 sec/batch\n",
      "Epoch: 1/20...  Training Step: 118...  Training loss: 2.9758...  6.2383 sec/batch\n",
      "Epoch: 1/20...  Training Step: 119...  Training loss: 2.9742...  6.2293 sec/batch\n",
      "Epoch: 1/20...  Training Step: 120...  Training loss: 2.9788...  6.3014 sec/batch\n",
      "Epoch: 1/20...  Training Step: 121...  Training loss: 2.9991...  6.1959 sec/batch\n",
      "Epoch: 1/20...  Training Step: 122...  Training loss: 2.9729...  6.2132 sec/batch\n",
      "Epoch: 1/20...  Training Step: 123...  Training loss: 2.9723...  6.2239 sec/batch\n",
      "Epoch: 1/20...  Training Step: 124...  Training loss: 2.9640...  6.2711 sec/batch\n",
      "Epoch: 1/20...  Training Step: 125...  Training loss: 2.9560...  6.2309 sec/batch\n",
      "Epoch: 1/20...  Training Step: 126...  Training loss: 3.0354...  6.2049 sec/batch\n",
      "Epoch: 1/20...  Training Step: 127...  Training loss: 3.0116...  6.2487 sec/batch\n",
      "Epoch: 1/20...  Training Step: 128...  Training loss: 3.0087...  6.3032 sec/batch\n",
      "Epoch: 1/20...  Training Step: 129...  Training loss: 2.9746...  6.2282 sec/batch\n",
      "Epoch: 1/20...  Training Step: 130...  Training loss: 2.9567...  6.2574 sec/batch\n",
      "Epoch: 1/20...  Training Step: 131...  Training loss: 2.9750...  6.3097 sec/batch\n",
      "Epoch: 1/20...  Training Step: 132...  Training loss: 2.9943...  6.2803 sec/batch\n",
      "Epoch: 1/20...  Training Step: 133...  Training loss: 2.9582...  6.2305 sec/batch\n",
      "Epoch: 1/20...  Training Step: 134...  Training loss: 2.9498...  6.2087 sec/batch\n",
      "Epoch: 1/20...  Training Step: 135...  Training loss: 2.9553...  6.1907 sec/batch\n",
      "Epoch: 1/20...  Training Step: 136...  Training loss: 2.9145...  6.2576 sec/batch\n",
      "Epoch: 1/20...  Training Step: 137...  Training loss: 2.9358...  6.2412 sec/batch\n",
      "Epoch: 1/20...  Training Step: 138...  Training loss: 2.9251...  6.2079 sec/batch\n",
      "Epoch: 1/20...  Training Step: 139...  Training loss: 2.9460...  6.2030 sec/batch\n",
      "Epoch: 1/20...  Training Step: 140...  Training loss: 2.9053...  6.2960 sec/batch\n",
      "Epoch: 1/20...  Training Step: 141...  Training loss: 2.9162...  6.2290 sec/batch\n",
      "Epoch: 1/20...  Training Step: 142...  Training loss: 2.9083...  6.2931 sec/batch\n",
      "Epoch: 1/20...  Training Step: 143...  Training loss: 2.8761...  6.2548 sec/batch\n",
      "Epoch: 1/20...  Training Step: 144...  Training loss: 2.8777...  6.2460 sec/batch\n",
      "Epoch: 1/20...  Training Step: 145...  Training loss: 2.8875...  6.2544 sec/batch\n",
      "Epoch: 1/20...  Training Step: 146...  Training loss: 2.8880...  6.2320 sec/batch\n",
      "Epoch: 1/20...  Training Step: 147...  Training loss: 2.8803...  6.2624 sec/batch\n",
      "Epoch: 1/20...  Training Step: 148...  Training loss: 2.8420...  6.2690 sec/batch\n",
      "Epoch: 1/20...  Training Step: 149...  Training loss: 2.8466...  6.2515 sec/batch\n",
      "Epoch: 1/20...  Training Step: 150...  Training loss: 2.8245...  6.2255 sec/batch\n",
      "Epoch: 1/20...  Training Step: 151...  Training loss: 2.8341...  6.2314 sec/batch\n",
      "Epoch: 1/20...  Training Step: 152...  Training loss: 2.7862...  6.2203 sec/batch\n",
      "Epoch: 1/20...  Training Step: 153...  Training loss: 2.7921...  6.2529 sec/batch\n",
      "Epoch: 1/20...  Training Step: 154...  Training loss: 2.7806...  6.3027 sec/batch\n",
      "Epoch: 1/20...  Training Step: 155...  Training loss: 2.7813...  6.2063 sec/batch\n",
      "Epoch: 1/20...  Training Step: 156...  Training loss: 2.7828...  6.2280 sec/batch\n",
      "Epoch: 1/20...  Training Step: 157...  Training loss: 2.7823...  6.2544 sec/batch\n",
      "Epoch: 1/20...  Training Step: 158...  Training loss: 2.7664...  6.2489 sec/batch\n",
      "Epoch: 1/20...  Training Step: 159...  Training loss: 2.7577...  6.2914 sec/batch\n",
      "Epoch: 1/20...  Training Step: 160...  Training loss: 2.7438...  6.3056 sec/batch\n",
      "Epoch: 1/20...  Training Step: 161...  Training loss: 2.7187...  6.2728 sec/batch\n",
      "Epoch: 1/20...  Training Step: 162...  Training loss: 2.7975...  6.2046 sec/batch\n",
      "Epoch: 1/20...  Training Step: 163...  Training loss: 2.7727...  6.2119 sec/batch\n",
      "Epoch: 1/20...  Training Step: 164...  Training loss: 2.8058...  6.2794 sec/batch\n",
      "Epoch: 1/20...  Training Step: 165...  Training loss: 2.7702...  6.2524 sec/batch\n",
      "Epoch: 1/20...  Training Step: 166...  Training loss: 2.7563...  6.2061 sec/batch\n",
      "Epoch: 1/20...  Training Step: 167...  Training loss: 2.7407...  6.2487 sec/batch\n",
      "Epoch: 1/20...  Training Step: 168...  Training loss: 2.7064...  6.2185 sec/batch\n",
      "Epoch: 1/20...  Training Step: 169...  Training loss: 2.7033...  6.2547 sec/batch\n",
      "Epoch: 1/20...  Training Step: 170...  Training loss: 2.7091...  6.2549 sec/batch\n",
      "Epoch: 1/20...  Training Step: 171...  Training loss: 2.6841...  6.2527 sec/batch\n",
      "Epoch: 1/20...  Training Step: 172...  Training loss: 2.6918...  6.2429 sec/batch\n",
      "Epoch: 1/20...  Training Step: 173...  Training loss: 2.6589...  6.2355 sec/batch\n",
      "Epoch: 1/20...  Training Step: 174...  Training loss: 2.6763...  6.2797 sec/batch\n",
      "Epoch: 1/20...  Training Step: 175...  Training loss: 2.6604...  6.2426 sec/batch\n",
      "Epoch: 1/20...  Training Step: 176...  Training loss: 2.6437...  6.3103 sec/batch\n",
      "Epoch: 1/20...  Training Step: 177...  Training loss: 2.6263...  6.2243 sec/batch\n",
      "Epoch: 1/20...  Training Step: 178...  Training loss: 2.6357...  6.2234 sec/batch\n",
      "Epoch: 1/20...  Training Step: 179...  Training loss: 2.7207...  6.2662 sec/batch\n",
      "Epoch: 1/20...  Training Step: 180...  Training loss: 2.5932...  6.1948 sec/batch\n",
      "Epoch: 1/20...  Training Step: 181...  Training loss: 2.5796...  6.2012 sec/batch\n",
      "Epoch: 1/20...  Training Step: 182...  Training loss: 2.5930...  6.1742 sec/batch\n",
      "Epoch: 1/20...  Training Step: 183...  Training loss: 2.6000...  6.2506 sec/batch\n",
      "Epoch: 1/20...  Training Step: 184...  Training loss: 2.5787...  6.1987 sec/batch\n",
      "Epoch: 1/20...  Training Step: 185...  Training loss: 2.5902...  6.2497 sec/batch\n",
      "Epoch: 1/20...  Training Step: 186...  Training loss: 2.5823...  6.1940 sec/batch\n",
      "Epoch: 1/20...  Training Step: 187...  Training loss: 2.5953...  6.2565 sec/batch\n",
      "Epoch: 1/20...  Training Step: 188...  Training loss: 2.5710...  6.2731 sec/batch\n",
      "Epoch: 1/20...  Training Step: 189...  Training loss: 2.5502...  6.2240 sec/batch\n",
      "Epoch: 1/20...  Training Step: 190...  Training loss: 2.5581...  6.2728 sec/batch\n",
      "Epoch: 1/20...  Training Step: 191...  Training loss: 2.5290...  6.2857 sec/batch\n",
      "Epoch: 1/20...  Training Step: 192...  Training loss: 2.5202...  6.1858 sec/batch\n",
      "Epoch: 1/20...  Training Step: 193...  Training loss: 2.5284...  6.2106 sec/batch\n",
      "Epoch: 1/20...  Training Step: 194...  Training loss: 2.5022...  6.2118 sec/batch\n",
      "Epoch: 1/20...  Training Step: 195...  Training loss: 2.5104...  6.2534 sec/batch\n",
      "Epoch: 1/20...  Training Step: 196...  Training loss: 2.5071...  6.2499 sec/batch\n",
      "Epoch: 1/20...  Training Step: 197...  Training loss: 2.5167...  6.2567 sec/batch\n",
      "Epoch: 1/20...  Training Step: 198...  Training loss: 2.5209...  6.2459 sec/batch\n",
      "Epoch: 1/20...  Training Step: 199...  Training loss: 2.4984...  6.2064 sec/batch\n",
      "Epoch: 1/20...  Training Step: 200...  Training loss: 2.4923...  6.2278 sec/batch\n",
      "Epoch: 1/20...  Training Step: 201...  Training loss: 2.4906...  6.2296 sec/batch\n",
      "Epoch: 1/20...  Training Step: 202...  Training loss: 2.4717...  6.3131 sec/batch\n",
      "Epoch: 1/20...  Training Step: 203...  Training loss: 2.4825...  6.2185 sec/batch\n",
      "Epoch: 1/20...  Training Step: 204...  Training loss: 2.4559...  6.2924 sec/batch\n",
      "Epoch: 1/20...  Training Step: 205...  Training loss: 2.4437...  6.2902 sec/batch\n",
      "Epoch: 1/20...  Training Step: 206...  Training loss: 2.4692...  6.2665 sec/batch\n",
      "Epoch: 1/20...  Training Step: 207...  Training loss: 2.4456...  6.2027 sec/batch\n",
      "Epoch: 2/20...  Training Step: 208...  Training loss: 2.4912...  6.2332 sec/batch\n",
      "Epoch: 2/20...  Training Step: 209...  Training loss: 2.4312...  6.2543 sec/batch\n",
      "Epoch: 2/20...  Training Step: 210...  Training loss: 2.4528...  6.2692 sec/batch\n",
      "Epoch: 2/20...  Training Step: 211...  Training loss: 2.4575...  6.1777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 212...  Training loss: 2.4163...  6.2847 sec/batch\n",
      "Epoch: 2/20...  Training Step: 213...  Training loss: 2.4238...  6.2231 sec/batch\n",
      "Epoch: 2/20...  Training Step: 214...  Training loss: 2.4039...  6.2314 sec/batch\n",
      "Epoch: 2/20...  Training Step: 215...  Training loss: 2.4120...  6.2356 sec/batch\n",
      "Epoch: 2/20...  Training Step: 216...  Training loss: 2.3855...  6.2332 sec/batch\n",
      "Epoch: 2/20...  Training Step: 217...  Training loss: 2.3958...  6.2768 sec/batch\n",
      "Epoch: 2/20...  Training Step: 218...  Training loss: 2.3943...  6.2717 sec/batch\n",
      "Epoch: 2/20...  Training Step: 219...  Training loss: 2.3660...  6.2455 sec/batch\n",
      "Epoch: 2/20...  Training Step: 220...  Training loss: 2.3913...  6.2467 sec/batch\n",
      "Epoch: 2/20...  Training Step: 221...  Training loss: 2.3730...  6.2034 sec/batch\n",
      "Epoch: 2/20...  Training Step: 222...  Training loss: 2.3512...  6.2488 sec/batch\n",
      "Epoch: 2/20...  Training Step: 223...  Training loss: 2.3826...  6.2099 sec/batch\n",
      "Epoch: 2/20...  Training Step: 224...  Training loss: 2.3618...  6.2288 sec/batch\n",
      "Epoch: 2/20...  Training Step: 225...  Training loss: 2.3617...  6.1947 sec/batch\n",
      "Epoch: 2/20...  Training Step: 226...  Training loss: 2.4169...  6.1984 sec/batch\n",
      "Epoch: 2/20...  Training Step: 227...  Training loss: 2.3748...  6.2582 sec/batch\n",
      "Epoch: 2/20...  Training Step: 228...  Training loss: 2.3420...  6.2126 sec/batch\n",
      "Epoch: 2/20...  Training Step: 229...  Training loss: 2.3814...  6.2209 sec/batch\n",
      "Epoch: 2/20...  Training Step: 230...  Training loss: 2.3213...  6.2534 sec/batch\n",
      "Epoch: 2/20...  Training Step: 231...  Training loss: 2.3511...  6.2790 sec/batch\n",
      "Epoch: 2/20...  Training Step: 232...  Training loss: 2.3434...  6.2743 sec/batch\n",
      "Epoch: 2/20...  Training Step: 233...  Training loss: 2.3628...  6.2604 sec/batch\n",
      "Epoch: 2/20...  Training Step: 234...  Training loss: 2.3544...  6.2716 sec/batch\n",
      "Epoch: 2/20...  Training Step: 235...  Training loss: 2.3286...  6.2358 sec/batch\n",
      "Epoch: 2/20...  Training Step: 236...  Training loss: 2.3330...  6.1834 sec/batch\n",
      "Epoch: 2/20...  Training Step: 237...  Training loss: 2.3481...  6.2133 sec/batch\n",
      "Epoch: 2/20...  Training Step: 238...  Training loss: 2.3041...  6.2577 sec/batch\n",
      "Epoch: 2/20...  Training Step: 239...  Training loss: 2.3554...  6.2804 sec/batch\n",
      "Epoch: 2/20...  Training Step: 240...  Training loss: 2.3403...  6.1892 sec/batch\n",
      "Epoch: 2/20...  Training Step: 241...  Training loss: 2.3250...  6.2577 sec/batch\n",
      "Epoch: 2/20...  Training Step: 242...  Training loss: 2.3208...  6.2484 sec/batch\n",
      "Epoch: 2/20...  Training Step: 243...  Training loss: 2.3063...  6.2452 sec/batch\n",
      "Epoch: 2/20...  Training Step: 244...  Training loss: 2.3193...  6.2273 sec/batch\n",
      "Epoch: 2/20...  Training Step: 245...  Training loss: 2.3046...  6.2338 sec/batch\n",
      "Epoch: 2/20...  Training Step: 246...  Training loss: 2.3334...  6.2476 sec/batch\n",
      "Epoch: 2/20...  Training Step: 247...  Training loss: 2.3297...  6.2304 sec/batch\n",
      "Epoch: 2/20...  Training Step: 248...  Training loss: 2.3271...  6.2360 sec/batch\n",
      "Epoch: 2/20...  Training Step: 249...  Training loss: 2.3029...  6.2714 sec/batch\n",
      "Epoch: 2/20...  Training Step: 250...  Training loss: 2.3157...  6.2178 sec/batch\n",
      "Epoch: 2/20...  Training Step: 251...  Training loss: 2.3439...  6.2112 sec/batch\n",
      "Epoch: 2/20...  Training Step: 252...  Training loss: 2.3037...  6.1796 sec/batch\n",
      "Epoch: 2/20...  Training Step: 253...  Training loss: 2.3066...  6.2093 sec/batch\n",
      "Epoch: 2/20...  Training Step: 254...  Training loss: 2.2888...  6.1949 sec/batch\n",
      "Epoch: 2/20...  Training Step: 255...  Training loss: 2.2963...  6.3165 sec/batch\n",
      "Epoch: 2/20...  Training Step: 256...  Training loss: 2.2992...  6.2337 sec/batch\n",
      "Epoch: 2/20...  Training Step: 257...  Training loss: 2.3214...  6.2644 sec/batch\n",
      "Epoch: 2/20...  Training Step: 258...  Training loss: 2.2927...  6.2393 sec/batch\n",
      "Epoch: 2/20...  Training Step: 259...  Training loss: 2.3012...  6.2895 sec/batch\n",
      "Epoch: 2/20...  Training Step: 260...  Training loss: 2.2959...  6.3141 sec/batch\n",
      "Epoch: 2/20...  Training Step: 261...  Training loss: 2.2927...  6.2592 sec/batch\n",
      "Epoch: 2/20...  Training Step: 262...  Training loss: 2.2737...  6.2460 sec/batch\n",
      "Epoch: 2/20...  Training Step: 263...  Training loss: 2.2837...  6.2391 sec/batch\n",
      "Epoch: 2/20...  Training Step: 264...  Training loss: 2.2489...  6.2455 sec/batch\n",
      "Epoch: 2/20...  Training Step: 265...  Training loss: 2.2709...  6.2208 sec/batch\n",
      "Epoch: 2/20...  Training Step: 266...  Training loss: 2.2800...  6.2939 sec/batch\n",
      "Epoch: 2/20...  Training Step: 267...  Training loss: 2.2883...  6.2092 sec/batch\n",
      "Epoch: 2/20...  Training Step: 268...  Training loss: 2.2570...  6.2211 sec/batch\n",
      "Epoch: 2/20...  Training Step: 269...  Training loss: 2.2507...  6.1865 sec/batch\n",
      "Epoch: 2/20...  Training Step: 270...  Training loss: 2.2805...  6.2456 sec/batch\n",
      "Epoch: 2/20...  Training Step: 271...  Training loss: 2.2570...  6.2115 sec/batch\n",
      "Epoch: 2/20...  Training Step: 272...  Training loss: 2.2564...  6.1980 sec/batch\n",
      "Epoch: 2/20...  Training Step: 273...  Training loss: 2.3036...  6.2114 sec/batch\n",
      "Epoch: 2/20...  Training Step: 274...  Training loss: 2.2642...  6.2467 sec/batch\n",
      "Epoch: 2/20...  Training Step: 275...  Training loss: 2.3085...  6.2543 sec/batch\n",
      "Epoch: 2/20...  Training Step: 276...  Training loss: 2.2747...  6.3164 sec/batch\n",
      "Epoch: 2/20...  Training Step: 277...  Training loss: 2.2636...  6.3148 sec/batch\n",
      "Epoch: 2/20...  Training Step: 278...  Training loss: 2.2757...  6.4541 sec/batch\n",
      "Epoch: 2/20...  Training Step: 279...  Training loss: 2.2585...  6.3539 sec/batch\n",
      "Epoch: 2/20...  Training Step: 280...  Training loss: 2.2722...  6.4899 sec/batch\n",
      "Epoch: 2/20...  Training Step: 281...  Training loss: 2.2569...  6.2426 sec/batch\n",
      "Epoch: 2/20...  Training Step: 282...  Training loss: 2.2505...  6.2011 sec/batch\n",
      "Epoch: 2/20...  Training Step: 283...  Training loss: 2.2292...  6.2691 sec/batch\n",
      "Epoch: 2/20...  Training Step: 284...  Training loss: 2.2607...  6.2092 sec/batch\n",
      "Epoch: 2/20...  Training Step: 285...  Training loss: 2.2245...  6.2170 sec/batch\n",
      "Epoch: 2/20...  Training Step: 286...  Training loss: 2.2639...  6.2432 sec/batch\n",
      "Epoch: 2/20...  Training Step: 287...  Training loss: 2.2434...  6.2460 sec/batch\n",
      "Epoch: 2/20...  Training Step: 288...  Training loss: 2.2305...  6.1842 sec/batch\n",
      "Epoch: 2/20...  Training Step: 289...  Training loss: 2.2388...  6.2468 sec/batch\n",
      "Epoch: 2/20...  Training Step: 290...  Training loss: 2.2216...  6.2534 sec/batch\n",
      "Epoch: 2/20...  Training Step: 291...  Training loss: 2.2342...  6.2317 sec/batch\n",
      "Epoch: 2/20...  Training Step: 292...  Training loss: 2.2290...  6.2651 sec/batch\n",
      "Epoch: 2/20...  Training Step: 293...  Training loss: 2.2191...  6.2301 sec/batch\n",
      "Epoch: 2/20...  Training Step: 294...  Training loss: 2.2096...  6.2472 sec/batch\n",
      "Epoch: 2/20...  Training Step: 295...  Training loss: 2.2164...  6.2630 sec/batch\n",
      "Epoch: 2/20...  Training Step: 296...  Training loss: 2.2419...  6.2140 sec/batch\n",
      "Epoch: 2/20...  Training Step: 297...  Training loss: 2.2243...  6.2046 sec/batch\n",
      "Epoch: 2/20...  Training Step: 298...  Training loss: 2.2297...  6.2215 sec/batch\n",
      "Epoch: 2/20...  Training Step: 299...  Training loss: 2.2177...  6.1919 sec/batch\n",
      "Epoch: 2/20...  Training Step: 300...  Training loss: 2.2067...  6.2032 sec/batch\n",
      "Epoch: 2/20...  Training Step: 301...  Training loss: 2.2086...  6.1938 sec/batch\n",
      "Epoch: 2/20...  Training Step: 302...  Training loss: 2.2239...  6.1981 sec/batch\n",
      "Epoch: 2/20...  Training Step: 303...  Training loss: 2.2193...  6.2073 sec/batch\n",
      "Epoch: 2/20...  Training Step: 304...  Training loss: 2.2187...  6.2901 sec/batch\n",
      "Epoch: 2/20...  Training Step: 305...  Training loss: 2.1917...  6.2519 sec/batch\n",
      "Epoch: 2/20...  Training Step: 306...  Training loss: 2.1899...  6.2860 sec/batch\n",
      "Epoch: 2/20...  Training Step: 307...  Training loss: 2.2206...  6.2489 sec/batch\n",
      "Epoch: 2/20...  Training Step: 308...  Training loss: 2.2136...  6.2106 sec/batch\n",
      "Epoch: 2/20...  Training Step: 309...  Training loss: 2.2026...  6.2615 sec/batch\n",
      "Epoch: 2/20...  Training Step: 310...  Training loss: 2.2135...  6.2226 sec/batch\n",
      "Epoch: 2/20...  Training Step: 311...  Training loss: 2.2434...  6.1972 sec/batch\n",
      "Epoch: 2/20...  Training Step: 312...  Training loss: 2.2142...  6.2472 sec/batch\n",
      "Epoch: 2/20...  Training Step: 313...  Training loss: 2.2134...  6.2365 sec/batch\n",
      "Epoch: 2/20...  Training Step: 314...  Training loss: 2.2057...  6.2179 sec/batch\n",
      "Epoch: 2/20...  Training Step: 315...  Training loss: 2.2138...  6.2796 sec/batch\n",
      "Epoch: 2/20...  Training Step: 316...  Training loss: 2.1902...  6.2144 sec/batch\n",
      "Epoch: 2/20...  Training Step: 317...  Training loss: 2.2411...  6.2430 sec/batch\n",
      "Epoch: 2/20...  Training Step: 318...  Training loss: 2.2297...  6.2182 sec/batch\n",
      "Epoch: 2/20...  Training Step: 319...  Training loss: 2.2005...  6.2269 sec/batch\n",
      "Epoch: 2/20...  Training Step: 320...  Training loss: 2.2070...  6.1954 sec/batch\n",
      "Epoch: 2/20...  Training Step: 321...  Training loss: 2.1896...  6.2806 sec/batch\n",
      "Epoch: 2/20...  Training Step: 322...  Training loss: 2.2131...  6.2232 sec/batch\n",
      "Epoch: 2/20...  Training Step: 323...  Training loss: 2.2008...  6.2556 sec/batch\n",
      "Epoch: 2/20...  Training Step: 324...  Training loss: 2.2068...  6.2361 sec/batch\n",
      "Epoch: 2/20...  Training Step: 325...  Training loss: 2.1984...  6.2228 sec/batch\n",
      "Epoch: 2/20...  Training Step: 326...  Training loss: 2.2038...  6.2804 sec/batch\n",
      "Epoch: 2/20...  Training Step: 327...  Training loss: 2.1854...  6.2286 sec/batch\n",
      "Epoch: 2/20...  Training Step: 328...  Training loss: 2.1773...  6.2107 sec/batch\n",
      "Epoch: 2/20...  Training Step: 329...  Training loss: 2.1863...  6.2378 sec/batch\n",
      "Epoch: 2/20...  Training Step: 330...  Training loss: 2.1823...  6.2559 sec/batch\n",
      "Epoch: 2/20...  Training Step: 331...  Training loss: 2.1952...  6.2136 sec/batch\n",
      "Epoch: 2/20...  Training Step: 332...  Training loss: 2.1683...  6.2545 sec/batch\n",
      "Epoch: 2/20...  Training Step: 333...  Training loss: 2.1617...  6.2294 sec/batch\n",
      "Epoch: 2/20...  Training Step: 334...  Training loss: 2.1738...  6.2816 sec/batch\n",
      "Epoch: 2/20...  Training Step: 335...  Training loss: 2.1664...  6.2973 sec/batch\n",
      "Epoch: 2/20...  Training Step: 336...  Training loss: 2.1630...  6.2466 sec/batch\n",
      "Epoch: 2/20...  Training Step: 337...  Training loss: 2.1909...  6.2509 sec/batch\n",
      "Epoch: 2/20...  Training Step: 338...  Training loss: 2.1840...  6.2906 sec/batch\n",
      "Epoch: 2/20...  Training Step: 339...  Training loss: 2.1691...  6.2820 sec/batch\n",
      "Epoch: 2/20...  Training Step: 340...  Training loss: 2.1628...  6.2352 sec/batch\n",
      "Epoch: 2/20...  Training Step: 341...  Training loss: 2.1712...  6.2107 sec/batch\n",
      "Epoch: 2/20...  Training Step: 342...  Training loss: 2.1688...  6.2553 sec/batch\n",
      "Epoch: 2/20...  Training Step: 343...  Training loss: 2.1480...  6.2161 sec/batch\n",
      "Epoch: 2/20...  Training Step: 344...  Training loss: 2.1676...  6.2763 sec/batch\n",
      "Epoch: 2/20...  Training Step: 345...  Training loss: 2.1576...  6.2733 sec/batch\n",
      "Epoch: 2/20...  Training Step: 346...  Training loss: 2.1752...  6.2241 sec/batch\n",
      "Epoch: 2/20...  Training Step: 347...  Training loss: 2.1720...  6.1896 sec/batch\n",
      "Epoch: 2/20...  Training Step: 348...  Training loss: 2.1452...  6.2282 sec/batch\n",
      "Epoch: 2/20...  Training Step: 349...  Training loss: 2.1526...  6.3134 sec/batch\n",
      "Epoch: 2/20...  Training Step: 350...  Training loss: 2.1425...  6.2772 sec/batch\n",
      "Epoch: 2/20...  Training Step: 351...  Training loss: 2.1491...  6.2943 sec/batch\n",
      "Epoch: 2/20...  Training Step: 352...  Training loss: 2.1688...  6.2587 sec/batch\n",
      "Epoch: 2/20...  Training Step: 353...  Training loss: 2.1458...  6.2508 sec/batch\n",
      "Epoch: 2/20...  Training Step: 354...  Training loss: 2.1484...  6.2535 sec/batch\n",
      "Epoch: 2/20...  Training Step: 355...  Training loss: 2.1680...  6.2267 sec/batch\n",
      "Epoch: 2/20...  Training Step: 356...  Training loss: 2.1599...  6.2471 sec/batch\n",
      "Epoch: 2/20...  Training Step: 357...  Training loss: 2.1333...  6.2073 sec/batch\n",
      "Epoch: 2/20...  Training Step: 358...  Training loss: 2.1622...  6.2574 sec/batch\n",
      "Epoch: 2/20...  Training Step: 359...  Training loss: 2.1149...  6.2032 sec/batch\n",
      "Epoch: 2/20...  Training Step: 360...  Training loss: 2.1349...  6.2522 sec/batch\n",
      "Epoch: 2/20...  Training Step: 361...  Training loss: 2.1368...  6.2255 sec/batch\n",
      "Epoch: 2/20...  Training Step: 362...  Training loss: 2.1371...  6.2553 sec/batch\n",
      "Epoch: 2/20...  Training Step: 363...  Training loss: 2.1411...  6.2680 sec/batch\n",
      "Epoch: 2/20...  Training Step: 364...  Training loss: 2.1360...  6.3416 sec/batch\n",
      "Epoch: 2/20...  Training Step: 365...  Training loss: 2.1312...  6.2260 sec/batch\n",
      "Epoch: 2/20...  Training Step: 366...  Training loss: 2.1513...  6.2275 sec/batch\n",
      "Epoch: 2/20...  Training Step: 367...  Training loss: 2.1355...  6.2376 sec/batch\n",
      "Epoch: 2/20...  Training Step: 368...  Training loss: 2.1235...  6.2546 sec/batch\n",
      "Epoch: 2/20...  Training Step: 369...  Training loss: 2.1133...  6.2158 sec/batch\n",
      "Epoch: 2/20...  Training Step: 370...  Training loss: 2.1317...  6.2608 sec/batch\n",
      "Epoch: 2/20...  Training Step: 371...  Training loss: 2.1669...  6.2070 sec/batch\n",
      "Epoch: 2/20...  Training Step: 372...  Training loss: 2.1871...  6.2325 sec/batch\n",
      "Epoch: 2/20...  Training Step: 373...  Training loss: 2.1510...  6.2040 sec/batch\n",
      "Epoch: 2/20...  Training Step: 374...  Training loss: 2.1385...  6.2288 sec/batch\n",
      "Epoch: 2/20...  Training Step: 375...  Training loss: 2.1115...  6.1853 sec/batch\n",
      "Epoch: 2/20...  Training Step: 376...  Training loss: 2.1241...  6.1864 sec/batch\n",
      "Epoch: 2/20...  Training Step: 377...  Training loss: 2.1304...  6.2370 sec/batch\n",
      "Epoch: 2/20...  Training Step: 378...  Training loss: 2.1528...  6.2530 sec/batch\n",
      "Epoch: 2/20...  Training Step: 379...  Training loss: 2.1525...  6.2629 sec/batch\n",
      "Epoch: 2/20...  Training Step: 380...  Training loss: 2.1359...  6.2732 sec/batch\n",
      "Epoch: 2/20...  Training Step: 381...  Training loss: 2.1524...  6.2891 sec/batch\n",
      "Epoch: 2/20...  Training Step: 382...  Training loss: 2.1162...  6.2403 sec/batch\n",
      "Epoch: 2/20...  Training Step: 383...  Training loss: 2.1052...  6.2057 sec/batch\n",
      "Epoch: 2/20...  Training Step: 384...  Training loss: 2.1150...  6.1788 sec/batch\n",
      "Epoch: 2/20...  Training Step: 385...  Training loss: 2.1016...  6.2555 sec/batch\n",
      "Epoch: 2/20...  Training Step: 386...  Training loss: 2.1355...  6.1776 sec/batch\n",
      "Epoch: 2/20...  Training Step: 387...  Training loss: 2.0994...  6.2186 sec/batch\n",
      "Epoch: 2/20...  Training Step: 388...  Training loss: 2.0840...  6.2435 sec/batch\n",
      "Epoch: 2/20...  Training Step: 389...  Training loss: 2.1292...  6.2379 sec/batch\n",
      "Epoch: 2/20...  Training Step: 390...  Training loss: 2.1209...  6.2478 sec/batch\n",
      "Epoch: 2/20...  Training Step: 391...  Training loss: 2.1107...  6.2622 sec/batch\n",
      "Epoch: 2/20...  Training Step: 392...  Training loss: 2.1390...  6.2532 sec/batch\n",
      "Epoch: 2/20...  Training Step: 393...  Training loss: 2.0956...  6.2440 sec/batch\n",
      "Epoch: 2/20...  Training Step: 394...  Training loss: 2.1192...  6.2675 sec/batch\n",
      "Epoch: 2/20...  Training Step: 395...  Training loss: 2.1156...  6.3375 sec/batch\n",
      "Epoch: 2/20...  Training Step: 396...  Training loss: 2.1058...  6.2777 sec/batch\n",
      "Epoch: 2/20...  Training Step: 397...  Training loss: 2.1243...  6.2221 sec/batch\n",
      "Epoch: 2/20...  Training Step: 398...  Training loss: 2.1015...  6.2447 sec/batch\n",
      "Epoch: 2/20...  Training Step: 399...  Training loss: 2.0986...  6.2875 sec/batch\n",
      "Epoch: 2/20...  Training Step: 400...  Training loss: 2.1160...  6.2642 sec/batch\n",
      "Epoch: 2/20...  Training Step: 401...  Training loss: 2.1036...  6.2451 sec/batch\n",
      "Epoch: 2/20...  Training Step: 402...  Training loss: 2.1053...  6.2619 sec/batch\n",
      "Epoch: 2/20...  Training Step: 403...  Training loss: 2.0971...  6.2282 sec/batch\n",
      "Epoch: 2/20...  Training Step: 404...  Training loss: 2.1235...  6.2430 sec/batch\n",
      "Epoch: 2/20...  Training Step: 405...  Training loss: 2.1159...  6.2734 sec/batch\n",
      "Epoch: 2/20...  Training Step: 406...  Training loss: 2.0978...  6.2510 sec/batch\n",
      "Epoch: 2/20...  Training Step: 407...  Training loss: 2.1022...  6.2995 sec/batch\n",
      "Epoch: 2/20...  Training Step: 408...  Training loss: 2.1157...  6.2562 sec/batch\n",
      "Epoch: 2/20...  Training Step: 409...  Training loss: 2.1092...  6.2257 sec/batch\n",
      "Epoch: 2/20...  Training Step: 410...  Training loss: 2.0949...  6.2275 sec/batch\n",
      "Epoch: 2/20...  Training Step: 411...  Training loss: 2.0869...  6.2240 sec/batch\n",
      "Epoch: 2/20...  Training Step: 412...  Training loss: 2.0840...  6.2374 sec/batch\n",
      "Epoch: 2/20...  Training Step: 413...  Training loss: 2.1214...  6.1922 sec/batch\n",
      "Epoch: 2/20...  Training Step: 414...  Training loss: 2.1028...  6.1760 sec/batch\n",
      "Epoch: 3/20...  Training Step: 415...  Training loss: 2.1654...  6.1992 sec/batch\n",
      "Epoch: 3/20...  Training Step: 416...  Training loss: 2.0894...  6.2389 sec/batch\n",
      "Epoch: 3/20...  Training Step: 417...  Training loss: 2.1115...  6.1995 sec/batch\n",
      "Epoch: 3/20...  Training Step: 418...  Training loss: 2.1319...  6.1779 sec/batch\n",
      "Epoch: 3/20...  Training Step: 419...  Training loss: 2.0914...  6.2169 sec/batch\n",
      "Epoch: 3/20...  Training Step: 420...  Training loss: 2.0976...  6.2249 sec/batch\n",
      "Epoch: 3/20...  Training Step: 421...  Training loss: 2.0914...  6.2571 sec/batch\n",
      "Epoch: 3/20...  Training Step: 422...  Training loss: 2.0990...  6.2945 sec/batch\n",
      "Epoch: 3/20...  Training Step: 423...  Training loss: 2.0715...  6.2912 sec/batch\n",
      "Epoch: 3/20...  Training Step: 424...  Training loss: 2.0935...  6.3320 sec/batch\n",
      "Epoch: 3/20...  Training Step: 425...  Training loss: 2.0882...  6.2534 sec/batch\n",
      "Epoch: 3/20...  Training Step: 426...  Training loss: 2.0681...  6.2361 sec/batch\n",
      "Epoch: 3/20...  Training Step: 427...  Training loss: 2.0921...  6.2392 sec/batch\n",
      "Epoch: 3/20...  Training Step: 428...  Training loss: 2.0838...  6.2499 sec/batch\n",
      "Epoch: 3/20...  Training Step: 429...  Training loss: 2.0646...  6.2427 sec/batch\n",
      "Epoch: 3/20...  Training Step: 430...  Training loss: 2.0840...  6.2373 sec/batch\n",
      "Epoch: 3/20...  Training Step: 431...  Training loss: 2.0590...  6.2012 sec/batch\n",
      "Epoch: 3/20...  Training Step: 432...  Training loss: 2.0676...  6.1657 sec/batch\n",
      "Epoch: 3/20...  Training Step: 433...  Training loss: 2.1086...  6.2517 sec/batch\n",
      "Epoch: 3/20...  Training Step: 434...  Training loss: 2.0828...  6.2492 sec/batch\n",
      "Epoch: 3/20...  Training Step: 435...  Training loss: 2.0491...  6.2939 sec/batch\n",
      "Epoch: 3/20...  Training Step: 436...  Training loss: 2.0887...  6.2007 sec/batch\n",
      "Epoch: 3/20...  Training Step: 437...  Training loss: 2.0352...  6.2489 sec/batch\n",
      "Epoch: 3/20...  Training Step: 438...  Training loss: 2.0664...  6.2398 sec/batch\n",
      "Epoch: 3/20...  Training Step: 439...  Training loss: 2.0445...  6.2566 sec/batch\n",
      "Epoch: 3/20...  Training Step: 440...  Training loss: 2.0875...  6.2307 sec/batch\n",
      "Epoch: 3/20...  Training Step: 441...  Training loss: 2.0728...  6.2170 sec/batch\n",
      "Epoch: 3/20...  Training Step: 442...  Training loss: 2.0460...  6.2138 sec/batch\n",
      "Epoch: 3/20...  Training Step: 443...  Training loss: 2.0494...  6.2743 sec/batch\n",
      "Epoch: 3/20...  Training Step: 444...  Training loss: 2.0801...  6.2067 sec/batch\n",
      "Epoch: 3/20...  Training Step: 445...  Training loss: 2.0391...  6.2263 sec/batch\n",
      "Epoch: 3/20...  Training Step: 446...  Training loss: 2.0752...  6.2868 sec/batch\n",
      "Epoch: 3/20...  Training Step: 447...  Training loss: 2.0654...  6.2375 sec/batch\n",
      "Epoch: 3/20...  Training Step: 448...  Training loss: 2.0435...  6.2740 sec/batch\n",
      "Epoch: 3/20...  Training Step: 449...  Training loss: 2.0590...  6.1978 sec/batch\n",
      "Epoch: 3/20...  Training Step: 450...  Training loss: 2.0443...  6.2608 sec/batch\n",
      "Epoch: 3/20...  Training Step: 451...  Training loss: 2.0454...  6.2047 sec/batch\n",
      "Epoch: 3/20...  Training Step: 452...  Training loss: 2.0328...  6.2332 sec/batch\n",
      "Epoch: 3/20...  Training Step: 453...  Training loss: 2.0548...  6.3054 sec/batch\n",
      "Epoch: 3/20...  Training Step: 454...  Training loss: 2.0549...  6.2734 sec/batch\n",
      "Epoch: 3/20...  Training Step: 455...  Training loss: 2.0513...  6.2735 sec/batch\n",
      "Epoch: 3/20...  Training Step: 456...  Training loss: 2.0303...  6.2426 sec/batch\n",
      "Epoch: 3/20...  Training Step: 457...  Training loss: 2.0388...  6.2200 sec/batch\n",
      "Epoch: 3/20...  Training Step: 458...  Training loss: 2.0587...  6.2180 sec/batch\n",
      "Epoch: 3/20...  Training Step: 459...  Training loss: 2.0217...  6.2832 sec/batch\n",
      "Epoch: 3/20...  Training Step: 460...  Training loss: 2.0368...  6.2453 sec/batch\n",
      "Epoch: 3/20...  Training Step: 461...  Training loss: 2.0349...  6.1914 sec/batch\n",
      "Epoch: 3/20...  Training Step: 462...  Training loss: 2.0327...  6.2743 sec/batch\n",
      "Epoch: 3/20...  Training Step: 463...  Training loss: 2.0323...  6.2350 sec/batch\n",
      "Epoch: 3/20...  Training Step: 464...  Training loss: 2.0603...  6.2542 sec/batch\n",
      "Epoch: 3/20...  Training Step: 465...  Training loss: 2.0347...  6.3101 sec/batch\n",
      "Epoch: 3/20...  Training Step: 466...  Training loss: 2.0225...  6.2581 sec/batch\n",
      "Epoch: 3/20...  Training Step: 467...  Training loss: 2.0340...  6.2912 sec/batch\n",
      "Epoch: 3/20...  Training Step: 468...  Training loss: 2.0456...  6.2354 sec/batch\n",
      "Epoch: 3/20...  Training Step: 469...  Training loss: 2.0180...  6.2001 sec/batch\n",
      "Epoch: 3/20...  Training Step: 470...  Training loss: 2.0192...  6.1875 sec/batch\n",
      "Epoch: 3/20...  Training Step: 471...  Training loss: 2.0121...  6.2744 sec/batch\n",
      "Epoch: 3/20...  Training Step: 472...  Training loss: 2.0113...  6.1968 sec/batch\n",
      "Epoch: 3/20...  Training Step: 473...  Training loss: 2.0123...  6.2468 sec/batch\n",
      "Epoch: 3/20...  Training Step: 474...  Training loss: 2.0156...  6.2204 sec/batch\n",
      "Epoch: 3/20...  Training Step: 475...  Training loss: 2.0012...  6.2065 sec/batch\n",
      "Epoch: 3/20...  Training Step: 476...  Training loss: 1.9933...  6.1961 sec/batch\n",
      "Epoch: 3/20...  Training Step: 477...  Training loss: 2.0350...  6.2720 sec/batch\n",
      "Epoch: 3/20...  Training Step: 478...  Training loss: 2.0042...  6.2921 sec/batch\n",
      "Epoch: 3/20...  Training Step: 479...  Training loss: 2.0254...  6.2318 sec/batch\n",
      "Epoch: 3/20...  Training Step: 480...  Training loss: 2.0696...  6.2813 sec/batch\n",
      "Epoch: 3/20...  Training Step: 481...  Training loss: 2.0190...  6.2123 sec/batch\n",
      "Epoch: 3/20...  Training Step: 482...  Training loss: 2.0603...  6.2964 sec/batch\n",
      "Epoch: 3/20...  Training Step: 483...  Training loss: 2.0231...  6.2801 sec/batch\n",
      "Epoch: 3/20...  Training Step: 484...  Training loss: 2.0199...  6.2212 sec/batch\n",
      "Epoch: 3/20...  Training Step: 485...  Training loss: 2.0355...  6.2784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 486...  Training loss: 2.0049...  6.2546 sec/batch\n",
      "Epoch: 3/20...  Training Step: 487...  Training loss: 2.0285...  6.2340 sec/batch\n",
      "Epoch: 3/20...  Training Step: 488...  Training loss: 2.0094...  6.2792 sec/batch\n",
      "Epoch: 3/20...  Training Step: 489...  Training loss: 1.9977...  6.2323 sec/batch\n",
      "Epoch: 3/20...  Training Step: 490...  Training loss: 2.0003...  6.3134 sec/batch\n",
      "Epoch: 3/20...  Training Step: 491...  Training loss: 2.0120...  6.2709 sec/batch\n",
      "Epoch: 3/20...  Training Step: 492...  Training loss: 1.9876...  6.2830 sec/batch\n",
      "Epoch: 3/20...  Training Step: 493...  Training loss: 2.0301...  6.2875 sec/batch\n",
      "Epoch: 3/20...  Training Step: 494...  Training loss: 2.0158...  6.2143 sec/batch\n",
      "Epoch: 3/20...  Training Step: 495...  Training loss: 2.0179...  6.3136 sec/batch\n",
      "Epoch: 3/20...  Training Step: 496...  Training loss: 2.0124...  6.2801 sec/batch\n",
      "Epoch: 3/20...  Training Step: 497...  Training loss: 2.0074...  6.2318 sec/batch\n",
      "Epoch: 3/20...  Training Step: 498...  Training loss: 2.0104...  6.2053 sec/batch\n",
      "Epoch: 3/20...  Training Step: 499...  Training loss: 1.9970...  6.1958 sec/batch\n",
      "Epoch: 3/20...  Training Step: 500...  Training loss: 1.9854...  6.2551 sec/batch\n",
      "Epoch: 3/20...  Training Step: 501...  Training loss: 1.9839...  6.2052 sec/batch\n",
      "Epoch: 3/20...  Training Step: 502...  Training loss: 1.9840...  6.2719 sec/batch\n",
      "Epoch: 3/20...  Training Step: 503...  Training loss: 2.0167...  6.2707 sec/batch\n",
      "Epoch: 3/20...  Training Step: 504...  Training loss: 1.9879...  6.2401 sec/batch\n",
      "Epoch: 3/20...  Training Step: 505...  Training loss: 2.0046...  6.2608 sec/batch\n",
      "Epoch: 3/20...  Training Step: 506...  Training loss: 1.9711...  6.2859 sec/batch\n",
      "Epoch: 3/20...  Training Step: 507...  Training loss: 1.9838...  6.3188 sec/batch\n",
      "Epoch: 3/20...  Training Step: 508...  Training loss: 1.9823...  6.3775 sec/batch\n",
      "Epoch: 3/20...  Training Step: 509...  Training loss: 1.9997...  6.4267 sec/batch\n",
      "Epoch: 3/20...  Training Step: 510...  Training loss: 1.9824...  6.4238 sec/batch\n",
      "Epoch: 3/20...  Training Step: 511...  Training loss: 2.0041...  6.3364 sec/batch\n",
      "Epoch: 3/20...  Training Step: 512...  Training loss: 1.9720...  6.2759 sec/batch\n",
      "Epoch: 3/20...  Training Step: 513...  Training loss: 1.9646...  6.2622 sec/batch\n",
      "Epoch: 3/20...  Training Step: 514...  Training loss: 1.9998...  6.2714 sec/batch\n",
      "Epoch: 3/20...  Training Step: 515...  Training loss: 1.9898...  6.2567 sec/batch\n",
      "Epoch: 3/20...  Training Step: 516...  Training loss: 1.9883...  6.2939 sec/batch\n",
      "Epoch: 3/20...  Training Step: 517...  Training loss: 1.9882...  6.2902 sec/batch\n",
      "Epoch: 3/20...  Training Step: 518...  Training loss: 2.0292...  6.2405 sec/batch\n",
      "Epoch: 3/20...  Training Step: 519...  Training loss: 1.9938...  6.2389 sec/batch\n",
      "Epoch: 3/20...  Training Step: 520...  Training loss: 1.9931...  6.2483 sec/batch\n",
      "Epoch: 3/20...  Training Step: 521...  Training loss: 1.9893...  6.3269 sec/batch\n",
      "Epoch: 3/20...  Training Step: 522...  Training loss: 1.9927...  6.3536 sec/batch\n",
      "Epoch: 3/20...  Training Step: 523...  Training loss: 1.9766...  6.3021 sec/batch\n",
      "Epoch: 3/20...  Training Step: 524...  Training loss: 2.0240...  6.2952 sec/batch\n",
      "Epoch: 3/20...  Training Step: 525...  Training loss: 2.0207...  6.3329 sec/batch\n",
      "Epoch: 3/20...  Training Step: 526...  Training loss: 1.9675...  6.3362 sec/batch\n",
      "Epoch: 3/20...  Training Step: 527...  Training loss: 1.9693...  6.3267 sec/batch\n",
      "Epoch: 3/20...  Training Step: 528...  Training loss: 1.9578...  6.3087 sec/batch\n",
      "Epoch: 3/20...  Training Step: 529...  Training loss: 1.9886...  6.2583 sec/batch\n",
      "Epoch: 3/20...  Training Step: 530...  Training loss: 1.9969...  6.3031 sec/batch\n",
      "Epoch: 3/20...  Training Step: 531...  Training loss: 1.9834...  6.3464 sec/batch\n",
      "Epoch: 3/20...  Training Step: 532...  Training loss: 1.9783...  6.2741 sec/batch\n",
      "Epoch: 3/20...  Training Step: 533...  Training loss: 1.9860...  6.2401 sec/batch\n",
      "Epoch: 3/20...  Training Step: 534...  Training loss: 1.9681...  6.2586 sec/batch\n",
      "Epoch: 3/20...  Training Step: 535...  Training loss: 1.9554...  6.2611 sec/batch\n",
      "Epoch: 3/20...  Training Step: 536...  Training loss: 1.9736...  6.2926 sec/batch\n",
      "Epoch: 3/20...  Training Step: 537...  Training loss: 1.9644...  6.2751 sec/batch\n",
      "Epoch: 3/20...  Training Step: 538...  Training loss: 1.9869...  6.3717 sec/batch\n",
      "Epoch: 3/20...  Training Step: 539...  Training loss: 1.9619...  6.3293 sec/batch\n",
      "Epoch: 3/20...  Training Step: 540...  Training loss: 1.9593...  6.2760 sec/batch\n",
      "Epoch: 3/20...  Training Step: 541...  Training loss: 1.9643...  6.2378 sec/batch\n",
      "Epoch: 3/20...  Training Step: 542...  Training loss: 1.9726...  6.2737 sec/batch\n",
      "Epoch: 3/20...  Training Step: 543...  Training loss: 1.9637...  6.1943 sec/batch\n",
      "Epoch: 3/20...  Training Step: 544...  Training loss: 1.9842...  6.2042 sec/batch\n",
      "Epoch: 3/20...  Training Step: 545...  Training loss: 1.9574...  6.2692 sec/batch\n",
      "Epoch: 3/20...  Training Step: 546...  Training loss: 1.9740...  6.2976 sec/batch\n",
      "Epoch: 3/20...  Training Step: 547...  Training loss: 1.9535...  6.2531 sec/batch\n",
      "Epoch: 3/20...  Training Step: 548...  Training loss: 1.9670...  6.2639 sec/batch\n",
      "Epoch: 3/20...  Training Step: 549...  Training loss: 1.9525...  6.2603 sec/batch\n",
      "Epoch: 3/20...  Training Step: 550...  Training loss: 1.9417...  6.2598 sec/batch\n",
      "Epoch: 3/20...  Training Step: 551...  Training loss: 1.9583...  6.2762 sec/batch\n",
      "Epoch: 3/20...  Training Step: 552...  Training loss: 1.9628...  6.3221 sec/batch\n",
      "Epoch: 3/20...  Training Step: 553...  Training loss: 1.9647...  6.2372 sec/batch\n",
      "Epoch: 3/20...  Training Step: 554...  Training loss: 1.9712...  6.2902 sec/batch\n",
      "Epoch: 3/20...  Training Step: 555...  Training loss: 1.9389...  6.3551 sec/batch\n",
      "Epoch: 3/20...  Training Step: 556...  Training loss: 1.9676...  6.2913 sec/batch\n",
      "Epoch: 3/20...  Training Step: 557...  Training loss: 1.9546...  6.3610 sec/batch\n",
      "Epoch: 3/20...  Training Step: 558...  Training loss: 1.9550...  6.2256 sec/batch\n",
      "Epoch: 3/20...  Training Step: 559...  Training loss: 1.9684...  6.2240 sec/batch\n",
      "Epoch: 3/20...  Training Step: 560...  Training loss: 1.9453...  6.2323 sec/batch\n",
      "Epoch: 3/20...  Training Step: 561...  Training loss: 1.9428...  6.2595 sec/batch\n",
      "Epoch: 3/20...  Training Step: 562...  Training loss: 1.9661...  6.2331 sec/batch\n",
      "Epoch: 3/20...  Training Step: 563...  Training loss: 1.9629...  6.2509 sec/batch\n",
      "Epoch: 3/20...  Training Step: 564...  Training loss: 1.9451...  6.2786 sec/batch\n",
      "Epoch: 3/20...  Training Step: 565...  Training loss: 1.9490...  6.2173 sec/batch\n",
      "Epoch: 3/20...  Training Step: 566...  Training loss: 1.9130...  6.2749 sec/batch\n",
      "Epoch: 3/20...  Training Step: 567...  Training loss: 1.9505...  6.3299 sec/batch\n",
      "Epoch: 3/20...  Training Step: 568...  Training loss: 1.9302...  6.2513 sec/batch\n",
      "Epoch: 3/20...  Training Step: 569...  Training loss: 1.9251...  6.2630 sec/batch\n",
      "Epoch: 3/20...  Training Step: 570...  Training loss: 1.9503...  6.2520 sec/batch\n",
      "Epoch: 3/20...  Training Step: 571...  Training loss: 1.9317...  6.2378 sec/batch\n",
      "Epoch: 3/20...  Training Step: 572...  Training loss: 1.9411...  6.2315 sec/batch\n",
      "Epoch: 3/20...  Training Step: 573...  Training loss: 1.9466...  6.2236 sec/batch\n",
      "Epoch: 3/20...  Training Step: 574...  Training loss: 1.9288...  6.2487 sec/batch\n",
      "Epoch: 3/20...  Training Step: 575...  Training loss: 1.9205...  6.2347 sec/batch\n",
      "Epoch: 3/20...  Training Step: 576...  Training loss: 1.9066...  6.2770 sec/batch\n",
      "Epoch: 3/20...  Training Step: 577...  Training loss: 1.9202...  6.2522 sec/batch\n",
      "Epoch: 3/20...  Training Step: 578...  Training loss: 1.9643...  6.2976 sec/batch\n",
      "Epoch: 3/20...  Training Step: 579...  Training loss: 1.9752...  6.2368 sec/batch\n",
      "Epoch: 3/20...  Training Step: 580...  Training loss: 1.9358...  6.2550 sec/batch\n",
      "Epoch: 3/20...  Training Step: 581...  Training loss: 1.9566...  6.3119 sec/batch\n",
      "Epoch: 3/20...  Training Step: 582...  Training loss: 1.9227...  6.2259 sec/batch\n",
      "Epoch: 3/20...  Training Step: 583...  Training loss: 1.9378...  6.2567 sec/batch\n",
      "Epoch: 3/20...  Training Step: 584...  Training loss: 1.9494...  6.2622 sec/batch\n",
      "Epoch: 3/20...  Training Step: 585...  Training loss: 1.9583...  6.2402 sec/batch\n",
      "Epoch: 3/20...  Training Step: 586...  Training loss: 1.9740...  6.2984 sec/batch\n",
      "Epoch: 3/20...  Training Step: 587...  Training loss: 1.9203...  6.2240 sec/batch\n",
      "Epoch: 3/20...  Training Step: 588...  Training loss: 1.9677...  6.2167 sec/batch\n",
      "Epoch: 3/20...  Training Step: 589...  Training loss: 1.9352...  6.3077 sec/batch\n",
      "Epoch: 3/20...  Training Step: 590...  Training loss: 1.9177...  6.2396 sec/batch\n",
      "Epoch: 3/20...  Training Step: 591...  Training loss: 1.9289...  6.2118 sec/batch\n",
      "Epoch: 3/20...  Training Step: 592...  Training loss: 1.9016...  6.2186 sec/batch\n",
      "Epoch: 3/20...  Training Step: 593...  Training loss: 1.9506...  6.2295 sec/batch\n",
      "Epoch: 3/20...  Training Step: 594...  Training loss: 1.9121...  6.2310 sec/batch\n",
      "Epoch: 3/20...  Training Step: 595...  Training loss: 1.8867...  6.2510 sec/batch\n",
      "Epoch: 3/20...  Training Step: 596...  Training loss: 1.9400...  6.2495 sec/batch\n",
      "Epoch: 3/20...  Training Step: 597...  Training loss: 1.9369...  6.2736 sec/batch\n",
      "Epoch: 3/20...  Training Step: 598...  Training loss: 1.9295...  6.2810 sec/batch\n",
      "Epoch: 3/20...  Training Step: 599...  Training loss: 1.9449...  6.2937 sec/batch\n",
      "Epoch: 3/20...  Training Step: 600...  Training loss: 1.9003...  6.2559 sec/batch\n",
      "Epoch: 3/20...  Training Step: 601...  Training loss: 1.9187...  6.2567 sec/batch\n",
      "Epoch: 3/20...  Training Step: 602...  Training loss: 1.9118...  6.2388 sec/batch\n",
      "Epoch: 3/20...  Training Step: 603...  Training loss: 1.9210...  6.2833 sec/batch\n",
      "Epoch: 3/20...  Training Step: 604...  Training loss: 1.9295...  6.2088 sec/batch\n",
      "Epoch: 3/20...  Training Step: 605...  Training loss: 1.9065...  6.2081 sec/batch\n",
      "Epoch: 3/20...  Training Step: 606...  Training loss: 1.9186...  6.2492 sec/batch\n",
      "Epoch: 3/20...  Training Step: 607...  Training loss: 1.9193...  6.2906 sec/batch\n",
      "Epoch: 3/20...  Training Step: 608...  Training loss: 1.9271...  6.3075 sec/batch\n",
      "Epoch: 3/20...  Training Step: 609...  Training loss: 1.9250...  6.2615 sec/batch\n",
      "Epoch: 3/20...  Training Step: 610...  Training loss: 1.9145...  6.2826 sec/batch\n",
      "Epoch: 3/20...  Training Step: 611...  Training loss: 1.9513...  6.3195 sec/batch\n",
      "Epoch: 3/20...  Training Step: 612...  Training loss: 1.9343...  6.2869 sec/batch\n",
      "Epoch: 3/20...  Training Step: 613...  Training loss: 1.9220...  6.3327 sec/batch\n",
      "Epoch: 3/20...  Training Step: 614...  Training loss: 1.9173...  6.2964 sec/batch\n",
      "Epoch: 3/20...  Training Step: 615...  Training loss: 1.9472...  6.2306 sec/batch\n",
      "Epoch: 3/20...  Training Step: 616...  Training loss: 1.9396...  6.2218 sec/batch\n",
      "Epoch: 3/20...  Training Step: 617...  Training loss: 1.9023...  6.2784 sec/batch\n",
      "Epoch: 3/20...  Training Step: 618...  Training loss: 1.9090...  6.2496 sec/batch\n",
      "Epoch: 3/20...  Training Step: 619...  Training loss: 1.9168...  6.2156 sec/batch\n",
      "Epoch: 3/20...  Training Step: 620...  Training loss: 1.9364...  6.2918 sec/batch\n",
      "Epoch: 3/20...  Training Step: 621...  Training loss: 1.9232...  6.2423 sec/batch\n",
      "Epoch: 4/20...  Training Step: 622...  Training loss: 1.9787...  6.3215 sec/batch\n",
      "Epoch: 4/20...  Training Step: 623...  Training loss: 1.9091...  6.2282 sec/batch\n",
      "Epoch: 4/20...  Training Step: 624...  Training loss: 1.9126...  6.2766 sec/batch\n",
      "Epoch: 4/20...  Training Step: 625...  Training loss: 1.9480...  6.2709 sec/batch\n",
      "Epoch: 4/20...  Training Step: 626...  Training loss: 1.9158...  6.2372 sec/batch\n",
      "Epoch: 4/20...  Training Step: 627...  Training loss: 1.9036...  6.3816 sec/batch\n",
      "Epoch: 4/20...  Training Step: 628...  Training loss: 1.9223...  6.3668 sec/batch\n",
      "Epoch: 4/20...  Training Step: 629...  Training loss: 1.9303...  6.2998 sec/batch\n",
      "Epoch: 4/20...  Training Step: 630...  Training loss: 1.9202...  6.3105 sec/batch\n",
      "Epoch: 4/20...  Training Step: 631...  Training loss: 1.9200...  6.2597 sec/batch\n",
      "Epoch: 4/20...  Training Step: 632...  Training loss: 1.9133...  6.2857 sec/batch\n",
      "Epoch: 4/20...  Training Step: 633...  Training loss: 1.8854...  6.2804 sec/batch\n",
      "Epoch: 4/20...  Training Step: 634...  Training loss: 1.9209...  6.2797 sec/batch\n",
      "Epoch: 4/20...  Training Step: 635...  Training loss: 1.9171...  6.2997 sec/batch\n",
      "Epoch: 4/20...  Training Step: 636...  Training loss: 1.8985...  6.2666 sec/batch\n",
      "Epoch: 4/20...  Training Step: 637...  Training loss: 1.8910...  6.2484 sec/batch\n",
      "Epoch: 4/20...  Training Step: 638...  Training loss: 1.8919...  6.2413 sec/batch\n",
      "Epoch: 4/20...  Training Step: 639...  Training loss: 1.8910...  6.3458 sec/batch\n",
      "Epoch: 4/20...  Training Step: 640...  Training loss: 1.9206...  6.2961 sec/batch\n",
      "Epoch: 4/20...  Training Step: 641...  Training loss: 1.9128...  6.2838 sec/batch\n",
      "Epoch: 4/20...  Training Step: 642...  Training loss: 1.8754...  6.2580 sec/batch\n",
      "Epoch: 4/20...  Training Step: 643...  Training loss: 1.9123...  6.2534 sec/batch\n",
      "Epoch: 4/20...  Training Step: 644...  Training loss: 1.8700...  6.1933 sec/batch\n",
      "Epoch: 4/20...  Training Step: 645...  Training loss: 1.9099...  6.2011 sec/batch\n",
      "Epoch: 4/20...  Training Step: 646...  Training loss: 1.8760...  6.2042 sec/batch\n",
      "Epoch: 4/20...  Training Step: 647...  Training loss: 1.9165...  6.2470 sec/batch\n",
      "Epoch: 4/20...  Training Step: 648...  Training loss: 1.8937...  6.2830 sec/batch\n",
      "Epoch: 4/20...  Training Step: 649...  Training loss: 1.8673...  6.2587 sec/batch\n",
      "Epoch: 4/20...  Training Step: 650...  Training loss: 1.8727...  6.1898 sec/batch\n",
      "Epoch: 4/20...  Training Step: 651...  Training loss: 1.9084...  6.2137 sec/batch\n",
      "Epoch: 4/20...  Training Step: 652...  Training loss: 1.8731...  6.2079 sec/batch\n",
      "Epoch: 4/20...  Training Step: 653...  Training loss: 1.9021...  6.2219 sec/batch\n",
      "Epoch: 4/20...  Training Step: 654...  Training loss: 1.8965...  6.2855 sec/batch\n",
      "Epoch: 4/20...  Training Step: 655...  Training loss: 1.8713...  6.2287 sec/batch\n",
      "Epoch: 4/20...  Training Step: 656...  Training loss: 1.8887...  6.2728 sec/batch\n",
      "Epoch: 4/20...  Training Step: 657...  Training loss: 1.8649...  6.2874 sec/batch\n",
      "Epoch: 4/20...  Training Step: 658...  Training loss: 1.8780...  6.2970 sec/batch\n",
      "Epoch: 4/20...  Training Step: 659...  Training loss: 1.8762...  6.2261 sec/batch\n",
      "Epoch: 4/20...  Training Step: 660...  Training loss: 1.8666...  6.2420 sec/batch\n",
      "Epoch: 4/20...  Training Step: 661...  Training loss: 1.8754...  6.2400 sec/batch\n",
      "Epoch: 4/20...  Training Step: 662...  Training loss: 1.8724...  6.2410 sec/batch\n",
      "Epoch: 4/20...  Training Step: 663...  Training loss: 1.8628...  6.2596 sec/batch\n",
      "Epoch: 4/20...  Training Step: 664...  Training loss: 1.8754...  6.2639 sec/batch\n",
      "Epoch: 4/20...  Training Step: 665...  Training loss: 1.8832...  6.2775 sec/batch\n",
      "Epoch: 4/20...  Training Step: 666...  Training loss: 1.8551...  6.1792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 667...  Training loss: 1.8680...  6.2263 sec/batch\n",
      "Epoch: 4/20...  Training Step: 668...  Training loss: 1.8714...  6.2851 sec/batch\n",
      "Epoch: 4/20...  Training Step: 669...  Training loss: 1.8668...  6.3216 sec/batch\n",
      "Epoch: 4/20...  Training Step: 670...  Training loss: 1.8703...  6.2684 sec/batch\n",
      "Epoch: 4/20...  Training Step: 671...  Training loss: 1.8984...  6.3250 sec/batch\n",
      "Epoch: 4/20...  Training Step: 672...  Training loss: 1.8739...  6.2479 sec/batch\n",
      "Epoch: 4/20...  Training Step: 673...  Training loss: 1.8525...  6.2565 sec/batch\n",
      "Epoch: 4/20...  Training Step: 674...  Training loss: 1.8615...  6.1988 sec/batch\n",
      "Epoch: 4/20...  Training Step: 675...  Training loss: 1.8777...  6.2080 sec/batch\n",
      "Epoch: 4/20...  Training Step: 676...  Training loss: 1.8547...  6.2791 sec/batch\n",
      "Epoch: 4/20...  Training Step: 677...  Training loss: 1.8533...  6.2136 sec/batch\n",
      "Epoch: 4/20...  Training Step: 678...  Training loss: 1.8523...  6.2539 sec/batch\n",
      "Epoch: 4/20...  Training Step: 679...  Training loss: 1.8466...  6.2440 sec/batch\n",
      "Epoch: 4/20...  Training Step: 680...  Training loss: 1.8721...  6.1929 sec/batch\n",
      "Epoch: 4/20...  Training Step: 681...  Training loss: 1.8543...  6.2300 sec/batch\n",
      "Epoch: 4/20...  Training Step: 682...  Training loss: 1.8469...  6.2764 sec/batch\n",
      "Epoch: 4/20...  Training Step: 683...  Training loss: 1.8287...  6.3015 sec/batch\n",
      "Epoch: 4/20...  Training Step: 684...  Training loss: 1.8722...  6.3033 sec/batch\n",
      "Epoch: 4/20...  Training Step: 685...  Training loss: 1.8536...  6.2670 sec/batch\n",
      "Epoch: 4/20...  Training Step: 686...  Training loss: 1.8696...  6.2596 sec/batch\n",
      "Epoch: 4/20...  Training Step: 687...  Training loss: 1.9207...  6.2772 sec/batch\n",
      "Epoch: 4/20...  Training Step: 688...  Training loss: 1.8664...  6.2804 sec/batch\n",
      "Epoch: 4/20...  Training Step: 689...  Training loss: 1.8972...  6.2695 sec/batch\n",
      "Epoch: 4/20...  Training Step: 690...  Training loss: 1.8715...  6.2588 sec/batch\n",
      "Epoch: 4/20...  Training Step: 691...  Training loss: 1.8573...  6.2446 sec/batch\n",
      "Epoch: 4/20...  Training Step: 692...  Training loss: 1.8789...  6.2440 sec/batch\n",
      "Epoch: 4/20...  Training Step: 693...  Training loss: 1.8450...  6.3093 sec/batch\n",
      "Epoch: 4/20...  Training Step: 694...  Training loss: 1.8662...  6.3070 sec/batch\n",
      "Epoch: 4/20...  Training Step: 695...  Training loss: 1.8511...  6.2485 sec/batch\n",
      "Epoch: 4/20...  Training Step: 696...  Training loss: 1.8316...  6.3280 sec/batch\n",
      "Epoch: 4/20...  Training Step: 697...  Training loss: 1.8447...  6.2964 sec/batch\n",
      "Epoch: 4/20...  Training Step: 698...  Training loss: 1.8548...  6.2545 sec/batch\n",
      "Epoch: 4/20...  Training Step: 699...  Training loss: 1.8289...  6.2475 sec/batch\n",
      "Epoch: 4/20...  Training Step: 700...  Training loss: 1.8665...  6.3629 sec/batch\n",
      "Epoch: 4/20...  Training Step: 701...  Training loss: 1.8586...  6.2128 sec/batch\n",
      "Epoch: 4/20...  Training Step: 702...  Training loss: 1.8656...  6.2869 sec/batch\n",
      "Epoch: 4/20...  Training Step: 703...  Training loss: 1.8673...  6.2394 sec/batch\n",
      "Epoch: 4/20...  Training Step: 704...  Training loss: 1.8429...  6.1885 sec/batch\n",
      "Epoch: 4/20...  Training Step: 705...  Training loss: 1.8565...  6.1840 sec/batch\n",
      "Epoch: 4/20...  Training Step: 706...  Training loss: 1.8412...  6.2217 sec/batch\n",
      "Epoch: 4/20...  Training Step: 707...  Training loss: 1.8352...  6.2696 sec/batch\n",
      "Epoch: 4/20...  Training Step: 708...  Training loss: 1.8349...  6.2554 sec/batch\n",
      "Epoch: 4/20...  Training Step: 709...  Training loss: 1.8478...  6.3006 sec/batch\n",
      "Epoch: 4/20...  Training Step: 710...  Training loss: 1.8623...  6.2476 sec/batch\n",
      "Epoch: 4/20...  Training Step: 711...  Training loss: 1.8375...  6.3157 sec/batch\n",
      "Epoch: 4/20...  Training Step: 712...  Training loss: 1.8553...  6.3394 sec/batch\n",
      "Epoch: 4/20...  Training Step: 713...  Training loss: 1.8210...  6.3057 sec/batch\n",
      "Epoch: 4/20...  Training Step: 714...  Training loss: 1.8516...  6.2792 sec/batch\n",
      "Epoch: 4/20...  Training Step: 715...  Training loss: 1.8301...  6.3105 sec/batch\n",
      "Epoch: 4/20...  Training Step: 716...  Training loss: 1.8484...  6.2802 sec/batch\n",
      "Epoch: 4/20...  Training Step: 717...  Training loss: 1.8227...  6.2417 sec/batch\n",
      "Epoch: 4/20...  Training Step: 718...  Training loss: 1.8636...  6.2465 sec/batch\n",
      "Epoch: 4/20...  Training Step: 719...  Training loss: 1.8049...  6.2225 sec/batch\n",
      "Epoch: 4/20...  Training Step: 720...  Training loss: 1.8064...  6.2592 sec/batch\n",
      "Epoch: 4/20...  Training Step: 721...  Training loss: 1.8568...  6.2705 sec/batch\n",
      "Epoch: 4/20...  Training Step: 722...  Training loss: 1.8455...  6.2597 sec/batch\n",
      "Epoch: 4/20...  Training Step: 723...  Training loss: 1.8506...  6.1994 sec/batch\n",
      "Epoch: 4/20...  Training Step: 724...  Training loss: 1.8267...  6.2199 sec/batch\n",
      "Epoch: 4/20...  Training Step: 725...  Training loss: 1.8719...  6.2330 sec/batch\n",
      "Epoch: 4/20...  Training Step: 726...  Training loss: 1.8417...  6.2871 sec/batch\n",
      "Epoch: 4/20...  Training Step: 727...  Training loss: 1.8298...  6.2926 sec/batch\n",
      "Epoch: 4/20...  Training Step: 728...  Training loss: 1.8436...  6.2593 sec/batch\n",
      "Epoch: 4/20...  Training Step: 729...  Training loss: 1.8427...  6.3103 sec/batch\n",
      "Epoch: 4/20...  Training Step: 730...  Training loss: 1.8233...  6.2795 sec/batch\n",
      "Epoch: 4/20...  Training Step: 731...  Training loss: 1.8852...  6.2872 sec/batch\n",
      "Epoch: 4/20...  Training Step: 732...  Training loss: 1.8730...  6.2608 sec/batch\n",
      "Epoch: 4/20...  Training Step: 733...  Training loss: 1.8165...  6.3151 sec/batch\n",
      "Epoch: 4/20...  Training Step: 734...  Training loss: 1.8181...  6.2802 sec/batch\n",
      "Epoch: 4/20...  Training Step: 735...  Training loss: 1.8234...  6.2497 sec/batch\n",
      "Epoch: 4/20...  Training Step: 736...  Training loss: 1.8411...  6.2574 sec/batch\n",
      "Epoch: 4/20...  Training Step: 737...  Training loss: 1.8509...  6.2534 sec/batch\n",
      "Epoch: 4/20...  Training Step: 738...  Training loss: 1.8399...  6.2033 sec/batch\n",
      "Epoch: 4/20...  Training Step: 739...  Training loss: 1.8405...  6.3275 sec/batch\n",
      "Epoch: 4/20...  Training Step: 740...  Training loss: 1.8477...  6.2665 sec/batch\n",
      "Epoch: 4/20...  Training Step: 741...  Training loss: 1.8260...  6.2928 sec/batch\n",
      "Epoch: 4/20...  Training Step: 742...  Training loss: 1.8037...  6.2940 sec/batch\n",
      "Epoch: 4/20...  Training Step: 743...  Training loss: 1.8318...  6.2805 sec/batch\n",
      "Epoch: 4/20...  Training Step: 744...  Training loss: 1.8054...  6.2761 sec/batch\n",
      "Epoch: 4/20...  Training Step: 745...  Training loss: 1.8306...  6.2823 sec/batch\n",
      "Epoch: 4/20...  Training Step: 746...  Training loss: 1.8206...  6.2900 sec/batch\n",
      "Epoch: 4/20...  Training Step: 747...  Training loss: 1.8061...  6.3070 sec/batch\n",
      "Epoch: 4/20...  Training Step: 748...  Training loss: 1.8018...  6.2399 sec/batch\n",
      "Epoch: 4/20...  Training Step: 749...  Training loss: 1.8257...  6.2854 sec/batch\n",
      "Epoch: 4/20...  Training Step: 750...  Training loss: 1.8285...  6.3006 sec/batch\n",
      "Epoch: 4/20...  Training Step: 751...  Training loss: 1.8535...  6.2615 sec/batch\n",
      "Epoch: 4/20...  Training Step: 752...  Training loss: 1.8129...  6.3286 sec/batch\n",
      "Epoch: 4/20...  Training Step: 753...  Training loss: 1.8259...  6.2629 sec/batch\n",
      "Epoch: 4/20...  Training Step: 754...  Training loss: 1.8172...  6.2314 sec/batch\n",
      "Epoch: 4/20...  Training Step: 755...  Training loss: 1.8081...  6.3245 sec/batch\n",
      "Epoch: 4/20...  Training Step: 756...  Training loss: 1.8037...  6.2625 sec/batch\n",
      "Epoch: 4/20...  Training Step: 757...  Training loss: 1.8067...  6.2825 sec/batch\n",
      "Epoch: 4/20...  Training Step: 758...  Training loss: 1.8042...  6.2317 sec/batch\n",
      "Epoch: 4/20...  Training Step: 759...  Training loss: 1.8208...  6.2207 sec/batch\n",
      "Epoch: 4/20...  Training Step: 760...  Training loss: 1.8200...  6.2138 sec/batch\n",
      "Epoch: 4/20...  Training Step: 761...  Training loss: 1.8115...  6.2127 sec/batch\n",
      "Epoch: 4/20...  Training Step: 762...  Training loss: 1.8013...  6.2256 sec/batch\n",
      "Epoch: 4/20...  Training Step: 763...  Training loss: 1.8174...  6.2266 sec/batch\n",
      "Epoch: 4/20...  Training Step: 764...  Training loss: 1.8212...  6.2672 sec/batch\n",
      "Epoch: 4/20...  Training Step: 765...  Training loss: 1.8066...  6.1866 sec/batch\n",
      "Epoch: 4/20...  Training Step: 766...  Training loss: 1.8199...  6.1901 sec/batch\n",
      "Epoch: 4/20...  Training Step: 767...  Training loss: 1.7964...  6.2285 sec/batch\n",
      "Epoch: 4/20...  Training Step: 768...  Training loss: 1.7907...  6.2395 sec/batch\n",
      "Epoch: 4/20...  Training Step: 769...  Training loss: 1.8224...  6.2534 sec/batch\n",
      "Epoch: 4/20...  Training Step: 770...  Training loss: 1.8202...  6.3156 sec/batch\n",
      "Epoch: 4/20...  Training Step: 771...  Training loss: 1.7929...  6.3377 sec/batch\n",
      "Epoch: 4/20...  Training Step: 772...  Training loss: 1.7996...  6.2976 sec/batch\n",
      "Epoch: 4/20...  Training Step: 773...  Training loss: 1.7711...  6.3574 sec/batch\n",
      "Epoch: 4/20...  Training Step: 774...  Training loss: 1.8012...  6.3060 sec/batch\n",
      "Epoch: 4/20...  Training Step: 775...  Training loss: 1.7824...  6.2806 sec/batch\n",
      "Epoch: 4/20...  Training Step: 776...  Training loss: 1.7873...  6.2926 sec/batch\n",
      "Epoch: 4/20...  Training Step: 777...  Training loss: 1.8086...  6.2441 sec/batch\n",
      "Epoch: 4/20...  Training Step: 778...  Training loss: 1.7854...  6.3102 sec/batch\n",
      "Epoch: 4/20...  Training Step: 779...  Training loss: 1.7993...  6.2537 sec/batch\n",
      "Epoch: 4/20...  Training Step: 780...  Training loss: 1.8024...  6.1887 sec/batch\n",
      "Epoch: 4/20...  Training Step: 781...  Training loss: 1.7859...  6.2600 sec/batch\n",
      "Epoch: 4/20...  Training Step: 782...  Training loss: 1.7896...  6.2720 sec/batch\n",
      "Epoch: 4/20...  Training Step: 783...  Training loss: 1.7761...  6.2284 sec/batch\n",
      "Epoch: 4/20...  Training Step: 784...  Training loss: 1.7856...  6.2153 sec/batch\n",
      "Epoch: 4/20...  Training Step: 785...  Training loss: 1.8257...  6.3133 sec/batch\n",
      "Epoch: 4/20...  Training Step: 786...  Training loss: 1.8357...  6.3222 sec/batch\n",
      "Epoch: 4/20...  Training Step: 787...  Training loss: 1.8051...  6.3227 sec/batch\n",
      "Epoch: 4/20...  Training Step: 788...  Training loss: 1.8193...  6.2683 sec/batch\n",
      "Epoch: 4/20...  Training Step: 789...  Training loss: 1.7845...  6.2392 sec/batch\n",
      "Epoch: 4/20...  Training Step: 790...  Training loss: 1.8108...  6.2491 sec/batch\n",
      "Epoch: 4/20...  Training Step: 791...  Training loss: 1.8112...  6.2747 sec/batch\n",
      "Epoch: 4/20...  Training Step: 792...  Training loss: 1.8224...  6.2455 sec/batch\n",
      "Epoch: 4/20...  Training Step: 793...  Training loss: 1.8291...  6.2646 sec/batch\n",
      "Epoch: 4/20...  Training Step: 794...  Training loss: 1.7903...  6.2729 sec/batch\n",
      "Epoch: 4/20...  Training Step: 795...  Training loss: 1.8377...  6.2352 sec/batch\n",
      "Epoch: 4/20...  Training Step: 796...  Training loss: 1.8047...  6.2069 sec/batch\n",
      "Epoch: 4/20...  Training Step: 797...  Training loss: 1.7866...  6.1761 sec/batch\n",
      "Epoch: 4/20...  Training Step: 798...  Training loss: 1.8093...  6.2366 sec/batch\n",
      "Epoch: 4/20...  Training Step: 799...  Training loss: 1.7664...  6.2756 sec/batch\n",
      "Epoch: 4/20...  Training Step: 800...  Training loss: 1.8333...  6.2130 sec/batch\n",
      "Epoch: 4/20...  Training Step: 801...  Training loss: 1.7830...  6.2638 sec/batch\n",
      "Epoch: 4/20...  Training Step: 802...  Training loss: 1.7438...  6.2843 sec/batch\n",
      "Epoch: 4/20...  Training Step: 803...  Training loss: 1.7928...  6.2250 sec/batch\n",
      "Epoch: 4/20...  Training Step: 804...  Training loss: 1.8012...  6.2557 sec/batch\n",
      "Epoch: 4/20...  Training Step: 805...  Training loss: 1.8073...  6.2472 sec/batch\n",
      "Epoch: 4/20...  Training Step: 806...  Training loss: 1.8126...  6.1989 sec/batch\n",
      "Epoch: 4/20...  Training Step: 807...  Training loss: 1.7831...  6.1712 sec/batch\n",
      "Epoch: 4/20...  Training Step: 808...  Training loss: 1.7969...  6.2137 sec/batch\n",
      "Epoch: 4/20...  Training Step: 809...  Training loss: 1.7747...  6.2039 sec/batch\n",
      "Epoch: 4/20...  Training Step: 810...  Training loss: 1.7957...  6.2190 sec/batch\n",
      "Epoch: 4/20...  Training Step: 811...  Training loss: 1.8109...  6.2075 sec/batch\n",
      "Epoch: 4/20...  Training Step: 812...  Training loss: 1.7796...  6.2656 sec/batch\n",
      "Epoch: 4/20...  Training Step: 813...  Training loss: 1.7996...  6.2850 sec/batch\n",
      "Epoch: 4/20...  Training Step: 814...  Training loss: 1.8050...  6.2133 sec/batch\n",
      "Epoch: 4/20...  Training Step: 815...  Training loss: 1.8055...  6.2775 sec/batch\n",
      "Epoch: 4/20...  Training Step: 816...  Training loss: 1.8065...  6.2697 sec/batch\n",
      "Epoch: 4/20...  Training Step: 817...  Training loss: 1.7901...  6.2259 sec/batch\n",
      "Epoch: 4/20...  Training Step: 818...  Training loss: 1.8386...  6.2532 sec/batch\n",
      "Epoch: 4/20...  Training Step: 819...  Training loss: 1.7984...  6.2387 sec/batch\n",
      "Epoch: 4/20...  Training Step: 820...  Training loss: 1.7816...  6.1922 sec/batch\n",
      "Epoch: 4/20...  Training Step: 821...  Training loss: 1.7819...  6.2402 sec/batch\n",
      "Epoch: 4/20...  Training Step: 822...  Training loss: 1.8129...  6.3455 sec/batch\n",
      "Epoch: 4/20...  Training Step: 823...  Training loss: 1.8072...  6.2949 sec/batch\n",
      "Epoch: 4/20...  Training Step: 824...  Training loss: 1.7806...  6.2051 sec/batch\n",
      "Epoch: 4/20...  Training Step: 825...  Training loss: 1.7768...  6.2654 sec/batch\n",
      "Epoch: 4/20...  Training Step: 826...  Training loss: 1.7983...  6.2012 sec/batch\n",
      "Epoch: 4/20...  Training Step: 827...  Training loss: 1.7974...  6.2444 sec/batch\n",
      "Epoch: 4/20...  Training Step: 828...  Training loss: 1.8072...  6.2976 sec/batch\n",
      "Epoch: 5/20...  Training Step: 829...  Training loss: 1.8504...  6.2351 sec/batch\n",
      "Epoch: 5/20...  Training Step: 830...  Training loss: 1.7912...  6.3767 sec/batch\n",
      "Epoch: 5/20...  Training Step: 831...  Training loss: 1.7886...  6.2776 sec/batch\n",
      "Epoch: 5/20...  Training Step: 832...  Training loss: 1.8245...  6.2970 sec/batch\n",
      "Epoch: 5/20...  Training Step: 833...  Training loss: 1.7883...  6.2202 sec/batch\n",
      "Epoch: 5/20...  Training Step: 834...  Training loss: 1.7784...  6.2830 sec/batch\n",
      "Epoch: 5/20...  Training Step: 835...  Training loss: 1.7912...  6.2436 sec/batch\n",
      "Epoch: 5/20...  Training Step: 836...  Training loss: 1.8052...  6.2114 sec/batch\n",
      "Epoch: 5/20...  Training Step: 837...  Training loss: 1.8009...  6.2703 sec/batch\n",
      "Epoch: 5/20...  Training Step: 838...  Training loss: 1.7865...  6.2197 sec/batch\n",
      "Epoch: 5/20...  Training Step: 839...  Training loss: 1.7799...  6.2424 sec/batch\n",
      "Epoch: 5/20...  Training Step: 840...  Training loss: 1.7676...  6.1952 sec/batch\n",
      "Epoch: 5/20...  Training Step: 841...  Training loss: 1.7868...  6.2686 sec/batch\n",
      "Epoch: 5/20...  Training Step: 842...  Training loss: 1.7912...  6.3155 sec/batch\n",
      "Epoch: 5/20...  Training Step: 843...  Training loss: 1.7746...  6.3292 sec/batch\n",
      "Epoch: 5/20...  Training Step: 844...  Training loss: 1.7643...  6.3340 sec/batch\n",
      "Epoch: 5/20...  Training Step: 845...  Training loss: 1.7549...  6.3173 sec/batch\n",
      "Epoch: 5/20...  Training Step: 846...  Training loss: 1.7692...  6.2768 sec/batch\n",
      "Epoch: 5/20...  Training Step: 847...  Training loss: 1.7908...  6.2722 sec/batch\n",
      "Epoch: 5/20...  Training Step: 848...  Training loss: 1.7812...  6.2467 sec/batch\n",
      "Epoch: 5/20...  Training Step: 849...  Training loss: 1.7597...  6.2114 sec/batch\n",
      "Epoch: 5/20...  Training Step: 850...  Training loss: 1.7800...  6.2305 sec/batch\n",
      "Epoch: 5/20...  Training Step: 851...  Training loss: 1.7541...  6.2678 sec/batch\n",
      "Epoch: 5/20...  Training Step: 852...  Training loss: 1.7793...  6.2422 sec/batch\n",
      "Epoch: 5/20...  Training Step: 853...  Training loss: 1.7522...  6.3388 sec/batch\n",
      "Epoch: 5/20...  Training Step: 854...  Training loss: 1.7852...  6.3154 sec/batch\n",
      "Epoch: 5/20...  Training Step: 855...  Training loss: 1.7730...  6.2567 sec/batch\n",
      "Epoch: 5/20...  Training Step: 856...  Training loss: 1.7450...  6.2973 sec/batch\n",
      "Epoch: 5/20...  Training Step: 857...  Training loss: 1.7472...  6.2903 sec/batch\n",
      "Epoch: 5/20...  Training Step: 858...  Training loss: 1.7848...  6.3267 sec/batch\n",
      "Epoch: 5/20...  Training Step: 859...  Training loss: 1.7441...  6.2417 sec/batch\n",
      "Epoch: 5/20...  Training Step: 860...  Training loss: 1.7884...  6.3225 sec/batch\n",
      "Epoch: 5/20...  Training Step: 861...  Training loss: 1.7823...  6.2919 sec/batch\n",
      "Epoch: 5/20...  Training Step: 862...  Training loss: 1.7641...  6.3045 sec/batch\n",
      "Epoch: 5/20...  Training Step: 863...  Training loss: 1.7679...  6.2201 sec/batch\n",
      "Epoch: 5/20...  Training Step: 864...  Training loss: 1.7470...  6.1968 sec/batch\n",
      "Epoch: 5/20...  Training Step: 865...  Training loss: 1.7592...  6.2022 sec/batch\n",
      "Epoch: 5/20...  Training Step: 866...  Training loss: 1.7518...  6.2187 sec/batch\n",
      "Epoch: 5/20...  Training Step: 867...  Training loss: 1.7475...  6.1994 sec/batch\n",
      "Epoch: 5/20...  Training Step: 868...  Training loss: 1.7659...  6.2686 sec/batch\n",
      "Epoch: 5/20...  Training Step: 869...  Training loss: 1.7569...  6.1951 sec/batch\n",
      "Epoch: 5/20...  Training Step: 870...  Training loss: 1.7341...  6.2366 sec/batch\n",
      "Epoch: 5/20...  Training Step: 871...  Training loss: 1.7471...  6.2620 sec/batch\n",
      "Epoch: 5/20...  Training Step: 872...  Training loss: 1.7623...  6.2899 sec/batch\n",
      "Epoch: 5/20...  Training Step: 873...  Training loss: 1.7341...  6.2462 sec/batch\n",
      "Epoch: 5/20...  Training Step: 874...  Training loss: 1.7463...  6.3081 sec/batch\n",
      "Epoch: 5/20...  Training Step: 875...  Training loss: 1.7511...  6.3130 sec/batch\n",
      "Epoch: 5/20...  Training Step: 876...  Training loss: 1.7573...  6.1950 sec/batch\n",
      "Epoch: 5/20...  Training Step: 877...  Training loss: 1.7448...  6.2138 sec/batch\n",
      "Epoch: 5/20...  Training Step: 878...  Training loss: 1.7670...  6.2218 sec/batch\n",
      "Epoch: 5/20...  Training Step: 879...  Training loss: 1.7406...  6.3111 sec/batch\n",
      "Epoch: 5/20...  Training Step: 880...  Training loss: 1.7335...  6.3031 sec/batch\n",
      "Epoch: 5/20...  Training Step: 881...  Training loss: 1.7410...  6.2818 sec/batch\n",
      "Epoch: 5/20...  Training Step: 882...  Training loss: 1.7570...  6.2626 sec/batch\n",
      "Epoch: 5/20...  Training Step: 883...  Training loss: 1.7293...  6.2867 sec/batch\n",
      "Epoch: 5/20...  Training Step: 884...  Training loss: 1.7497...  6.2995 sec/batch\n",
      "Epoch: 5/20...  Training Step: 885...  Training loss: 1.7298...  6.3226 sec/batch\n",
      "Epoch: 5/20...  Training Step: 886...  Training loss: 1.7292...  6.3360 sec/batch\n",
      "Epoch: 5/20...  Training Step: 887...  Training loss: 1.7440...  6.2785 sec/batch\n",
      "Epoch: 5/20...  Training Step: 888...  Training loss: 1.7377...  6.2828 sec/batch\n",
      "Epoch: 5/20...  Training Step: 889...  Training loss: 1.7380...  6.2883 sec/batch\n",
      "Epoch: 5/20...  Training Step: 890...  Training loss: 1.7262...  6.3120 sec/batch\n",
      "Epoch: 5/20...  Training Step: 891...  Training loss: 1.7611...  6.3495 sec/batch\n",
      "Epoch: 5/20...  Training Step: 892...  Training loss: 1.7330...  6.2370 sec/batch\n",
      "Epoch: 5/20...  Training Step: 893...  Training loss: 1.7617...  6.1890 sec/batch\n",
      "Epoch: 5/20...  Training Step: 894...  Training loss: 1.8021...  6.2010 sec/batch\n",
      "Epoch: 5/20...  Training Step: 895...  Training loss: 1.7510...  6.2384 sec/batch\n",
      "Epoch: 5/20...  Training Step: 896...  Training loss: 1.7722...  6.1986 sec/batch\n",
      "Epoch: 5/20...  Training Step: 897...  Training loss: 1.7537...  6.2357 sec/batch\n",
      "Epoch: 5/20...  Training Step: 898...  Training loss: 1.7494...  6.2583 sec/batch\n",
      "Epoch: 5/20...  Training Step: 899...  Training loss: 1.7601...  6.3151 sec/batch\n",
      "Epoch: 5/20...  Training Step: 900...  Training loss: 1.7195...  6.2416 sec/batch\n",
      "Epoch: 5/20...  Training Step: 901...  Training loss: 1.7532...  6.2628 sec/batch\n",
      "Epoch: 5/20...  Training Step: 902...  Training loss: 1.7346...  6.2363 sec/batch\n",
      "Epoch: 5/20...  Training Step: 903...  Training loss: 1.7210...  6.2457 sec/batch\n",
      "Epoch: 5/20...  Training Step: 904...  Training loss: 1.7322...  6.2017 sec/batch\n",
      "Epoch: 5/20...  Training Step: 905...  Training loss: 1.7333...  6.2328 sec/batch\n",
      "Epoch: 5/20...  Training Step: 906...  Training loss: 1.7158...  6.2423 sec/batch\n",
      "Epoch: 5/20...  Training Step: 907...  Training loss: 1.7495...  6.2546 sec/batch\n",
      "Epoch: 5/20...  Training Step: 908...  Training loss: 1.7510...  6.2891 sec/batch\n",
      "Epoch: 5/20...  Training Step: 909...  Training loss: 1.7434...  6.2316 sec/batch\n",
      "Epoch: 5/20...  Training Step: 910...  Training loss: 1.7459...  6.3379 sec/batch\n",
      "Epoch: 5/20...  Training Step: 911...  Training loss: 1.7469...  6.2977 sec/batch\n",
      "Epoch: 5/20...  Training Step: 912...  Training loss: 1.7386...  6.2474 sec/batch\n",
      "Epoch: 5/20...  Training Step: 913...  Training loss: 1.7386...  6.2798 sec/batch\n",
      "Epoch: 5/20...  Training Step: 914...  Training loss: 1.7161...  6.2440 sec/batch\n",
      "Epoch: 5/20...  Training Step: 915...  Training loss: 1.7285...  6.1908 sec/batch\n",
      "Epoch: 5/20...  Training Step: 916...  Training loss: 1.7381...  6.1702 sec/batch\n",
      "Epoch: 5/20...  Training Step: 917...  Training loss: 1.7509...  6.1952 sec/batch\n",
      "Epoch: 5/20...  Training Step: 918...  Training loss: 1.7275...  6.2720 sec/batch\n",
      "Epoch: 5/20...  Training Step: 919...  Training loss: 1.7445...  6.2466 sec/batch\n",
      "Epoch: 5/20...  Training Step: 920...  Training loss: 1.7057...  6.1725 sec/batch\n",
      "Epoch: 5/20...  Training Step: 921...  Training loss: 1.7444...  6.2009 sec/batch\n",
      "Epoch: 5/20...  Training Step: 922...  Training loss: 1.7136...  6.2562 sec/batch\n",
      "Epoch: 5/20...  Training Step: 923...  Training loss: 1.7365...  6.2233 sec/batch\n",
      "Epoch: 5/20...  Training Step: 924...  Training loss: 1.7152...  6.1822 sec/batch\n",
      "Epoch: 5/20...  Training Step: 925...  Training loss: 1.7529...  6.2370 sec/batch\n",
      "Epoch: 5/20...  Training Step: 926...  Training loss: 1.6949...  6.2404 sec/batch\n",
      "Epoch: 5/20...  Training Step: 927...  Training loss: 1.6926...  6.2817 sec/batch\n",
      "Epoch: 5/20...  Training Step: 928...  Training loss: 1.7554...  6.1769 sec/batch\n",
      "Epoch: 5/20...  Training Step: 929...  Training loss: 1.7294...  6.2005 sec/batch\n",
      "Epoch: 5/20...  Training Step: 930...  Training loss: 1.7375...  6.2371 sec/batch\n",
      "Epoch: 5/20...  Training Step: 931...  Training loss: 1.7090...  6.2990 sec/batch\n",
      "Epoch: 5/20...  Training Step: 932...  Training loss: 1.7479...  6.2927 sec/batch\n",
      "Epoch: 5/20...  Training Step: 933...  Training loss: 1.7381...  6.2037 sec/batch\n",
      "Epoch: 5/20...  Training Step: 934...  Training loss: 1.7166...  6.3103 sec/batch\n",
      "Epoch: 5/20...  Training Step: 935...  Training loss: 1.7362...  6.2508 sec/batch\n",
      "Epoch: 5/20...  Training Step: 936...  Training loss: 1.7330...  6.2411 sec/batch\n",
      "Epoch: 5/20...  Training Step: 937...  Training loss: 1.7088...  6.2309 sec/batch\n",
      "Epoch: 5/20...  Training Step: 938...  Training loss: 1.7664...  6.3417 sec/batch\n",
      "Epoch: 5/20...  Training Step: 939...  Training loss: 1.7656...  6.2651 sec/batch\n",
      "Epoch: 5/20...  Training Step: 940...  Training loss: 1.7221...  6.2161 sec/batch\n",
      "Epoch: 5/20...  Training Step: 941...  Training loss: 1.7073...  6.2643 sec/batch\n",
      "Epoch: 5/20...  Training Step: 942...  Training loss: 1.7244...  6.2593 sec/batch\n",
      "Epoch: 5/20...  Training Step: 943...  Training loss: 1.7345...  6.2468 sec/batch\n",
      "Epoch: 5/20...  Training Step: 944...  Training loss: 1.7474...  6.2515 sec/batch\n",
      "Epoch: 5/20...  Training Step: 945...  Training loss: 1.7411...  6.2661 sec/batch\n",
      "Epoch: 5/20...  Training Step: 946...  Training loss: 1.7483...  6.2429 sec/batch\n",
      "Epoch: 5/20...  Training Step: 947...  Training loss: 1.7364...  6.2371 sec/batch\n",
      "Epoch: 5/20...  Training Step: 948...  Training loss: 1.7194...  6.3110 sec/batch\n",
      "Epoch: 5/20...  Training Step: 949...  Training loss: 1.6974...  6.2437 sec/batch\n",
      "Epoch: 5/20...  Training Step: 950...  Training loss: 1.7173...  6.2420 sec/batch\n",
      "Epoch: 5/20...  Training Step: 951...  Training loss: 1.7015...  6.2274 sec/batch\n",
      "Epoch: 5/20...  Training Step: 952...  Training loss: 1.7237...  6.2300 sec/batch\n",
      "Epoch: 5/20...  Training Step: 953...  Training loss: 1.7205...  6.3272 sec/batch\n",
      "Epoch: 5/20...  Training Step: 954...  Training loss: 1.6937...  6.2107 sec/batch\n",
      "Epoch: 5/20...  Training Step: 955...  Training loss: 1.7084...  6.2513 sec/batch\n",
      "Epoch: 5/20...  Training Step: 956...  Training loss: 1.7190...  6.2073 sec/batch\n",
      "Epoch: 5/20...  Training Step: 957...  Training loss: 1.7260...  6.2133 sec/batch\n",
      "Epoch: 5/20...  Training Step: 958...  Training loss: 1.7466...  6.2885 sec/batch\n",
      "Epoch: 5/20...  Training Step: 959...  Training loss: 1.7119...  6.2474 sec/batch\n",
      "Epoch: 5/20...  Training Step: 960...  Training loss: 1.7231...  6.2151 sec/batch\n",
      "Epoch: 5/20...  Training Step: 961...  Training loss: 1.7068...  6.2302 sec/batch\n",
      "Epoch: 5/20...  Training Step: 962...  Training loss: 1.7143...  6.2360 sec/batch\n",
      "Epoch: 5/20...  Training Step: 963...  Training loss: 1.6879...  6.2328 sec/batch\n",
      "Epoch: 5/20...  Training Step: 964...  Training loss: 1.7074...  6.2154 sec/batch\n",
      "Epoch: 5/20...  Training Step: 965...  Training loss: 1.7017...  6.2650 sec/batch\n",
      "Epoch: 5/20...  Training Step: 966...  Training loss: 1.7065...  6.2283 sec/batch\n",
      "Epoch: 5/20...  Training Step: 967...  Training loss: 1.7170...  6.2230 sec/batch\n",
      "Epoch: 5/20...  Training Step: 968...  Training loss: 1.7090...  6.2404 sec/batch\n",
      "Epoch: 5/20...  Training Step: 969...  Training loss: 1.6980...  6.2352 sec/batch\n",
      "Epoch: 5/20...  Training Step: 970...  Training loss: 1.7020...  6.1857 sec/batch\n",
      "Epoch: 5/20...  Training Step: 971...  Training loss: 1.7126...  6.2137 sec/batch\n",
      "Epoch: 5/20...  Training Step: 972...  Training loss: 1.6953...  6.2271 sec/batch\n",
      "Epoch: 5/20...  Training Step: 973...  Training loss: 1.7128...  6.2116 sec/batch\n",
      "Epoch: 5/20...  Training Step: 974...  Training loss: 1.6926...  6.2118 sec/batch\n",
      "Epoch: 5/20...  Training Step: 975...  Training loss: 1.7011...  6.2599 sec/batch\n",
      "Epoch: 5/20...  Training Step: 976...  Training loss: 1.7142...  6.2249 sec/batch\n",
      "Epoch: 5/20...  Training Step: 977...  Training loss: 1.7094...  6.2123 sec/batch\n",
      "Epoch: 5/20...  Training Step: 978...  Training loss: 1.6851...  6.1989 sec/batch\n",
      "Epoch: 5/20...  Training Step: 979...  Training loss: 1.6976...  6.2191 sec/batch\n",
      "Epoch: 5/20...  Training Step: 980...  Training loss: 1.6610...  6.1584 sec/batch\n",
      "Epoch: 5/20...  Training Step: 981...  Training loss: 1.6987...  6.2911 sec/batch\n",
      "Epoch: 5/20...  Training Step: 982...  Training loss: 1.6729...  6.2119 sec/batch\n",
      "Epoch: 5/20...  Training Step: 983...  Training loss: 1.6796...  6.2072 sec/batch\n",
      "Epoch: 5/20...  Training Step: 984...  Training loss: 1.6929...  6.2879 sec/batch\n",
      "Epoch: 5/20...  Training Step: 985...  Training loss: 1.6867...  6.1549 sec/batch\n",
      "Epoch: 5/20...  Training Step: 986...  Training loss: 1.6974...  6.2483 sec/batch\n",
      "Epoch: 5/20...  Training Step: 987...  Training loss: 1.6813...  6.2019 sec/batch\n",
      "Epoch: 5/20...  Training Step: 988...  Training loss: 1.6790...  6.1965 sec/batch\n",
      "Epoch: 5/20...  Training Step: 989...  Training loss: 1.6671...  6.2063 sec/batch\n",
      "Epoch: 5/20...  Training Step: 990...  Training loss: 1.6565...  6.2474 sec/batch\n",
      "Epoch: 5/20...  Training Step: 991...  Training loss: 1.6829...  6.2409 sec/batch\n",
      "Epoch: 5/20...  Training Step: 992...  Training loss: 1.7159...  6.2448 sec/batch\n",
      "Epoch: 5/20...  Training Step: 993...  Training loss: 1.7268...  6.2725 sec/batch\n",
      "Epoch: 5/20...  Training Step: 994...  Training loss: 1.6876...  6.2176 sec/batch\n",
      "Epoch: 5/20...  Training Step: 995...  Training loss: 1.7122...  6.2974 sec/batch\n",
      "Epoch: 5/20...  Training Step: 996...  Training loss: 1.6766...  6.2978 sec/batch\n",
      "Epoch: 5/20...  Training Step: 997...  Training loss: 1.7144...  6.2414 sec/batch\n",
      "Epoch: 5/20...  Training Step: 998...  Training loss: 1.7177...  6.2400 sec/batch\n",
      "Epoch: 5/20...  Training Step: 999...  Training loss: 1.7106...  6.2299 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1000...  Training loss: 1.7310...  6.3046 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1001...  Training loss: 1.6882...  6.2525 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1002...  Training loss: 1.7485...  6.2009 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1003...  Training loss: 1.7025...  6.2577 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1004...  Training loss: 1.6834...  6.2824 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1005...  Training loss: 1.7093...  6.2419 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1006...  Training loss: 1.6559...  6.2496 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1007...  Training loss: 1.7218...  6.2372 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1008...  Training loss: 1.6789...  6.2253 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1009...  Training loss: 1.6474...  6.2608 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1010...  Training loss: 1.7002...  6.1968 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1011...  Training loss: 1.6798...  6.2309 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1012...  Training loss: 1.7130...  6.2824 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1013...  Training loss: 1.7027...  6.2508 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1014...  Training loss: 1.6727...  6.2590 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1015...  Training loss: 1.6835...  6.2114 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1016...  Training loss: 1.6789...  6.1678 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1017...  Training loss: 1.7040...  6.2350 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1018...  Training loss: 1.7244...  6.1996 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1019...  Training loss: 1.6885...  6.2198 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1020...  Training loss: 1.7100...  6.2889 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1021...  Training loss: 1.7017...  6.1987 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1022...  Training loss: 1.7072...  6.2738 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1023...  Training loss: 1.6981...  6.2602 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1024...  Training loss: 1.6855...  6.2329 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1025...  Training loss: 1.7422...  6.2235 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1026...  Training loss: 1.6967...  6.2637 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1027...  Training loss: 1.6896...  6.2416 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1028...  Training loss: 1.6835...  6.1841 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1029...  Training loss: 1.7176...  6.2463 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1030...  Training loss: 1.7265...  6.2868 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1031...  Training loss: 1.6742...  6.2606 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1032...  Training loss: 1.6881...  6.2069 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1033...  Training loss: 1.7148...  6.2310 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1034...  Training loss: 1.7034...  6.2826 sec/batch\n",
      "Epoch: 5/20...  Training Step: 1035...  Training loss: 1.7179...  6.2356 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1036...  Training loss: 1.7565...  6.2079 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1037...  Training loss: 1.6890...  6.2498 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1038...  Training loss: 1.6985...  6.2558 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1039...  Training loss: 1.7365...  6.2639 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1040...  Training loss: 1.6909...  6.2814 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1041...  Training loss: 1.6846...  6.2316 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1042...  Training loss: 1.6905...  6.3173 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1043...  Training loss: 1.7027...  6.2640 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1044...  Training loss: 1.7108...  6.2419 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1045...  Training loss: 1.6940...  6.3269 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1046...  Training loss: 1.6885...  6.2156 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1047...  Training loss: 1.6600...  6.1808 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1048...  Training loss: 1.6948...  6.1881 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1049...  Training loss: 1.6956...  6.1772 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1050...  Training loss: 1.6744...  6.2230 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1051...  Training loss: 1.6605...  6.2629 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1052...  Training loss: 1.6641...  6.1987 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1053...  Training loss: 1.6643...  6.2289 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1054...  Training loss: 1.6914...  6.2577 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1055...  Training loss: 1.6848...  6.1947 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1056...  Training loss: 1.6563...  6.2556 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1057...  Training loss: 1.6751...  6.2176 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1058...  Training loss: 1.6527...  6.2927 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1059...  Training loss: 1.6741...  6.2228 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1060...  Training loss: 1.6559...  6.1892 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1061...  Training loss: 1.6939...  6.2578 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1062...  Training loss: 1.6749...  6.2491 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1063...  Training loss: 1.6486...  6.2230 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1064...  Training loss: 1.6504...  6.2302 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1065...  Training loss: 1.6794...  6.2641 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1066...  Training loss: 1.6360...  6.2180 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1067...  Training loss: 1.6943...  6.2531 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1068...  Training loss: 1.6897...  6.1889 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1069...  Training loss: 1.6572...  6.2055 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1070...  Training loss: 1.6717...  6.2169 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1071...  Training loss: 1.6617...  6.2593 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1072...  Training loss: 1.6616...  6.2415 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1073...  Training loss: 1.6558...  6.2830 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1074...  Training loss: 1.6507...  6.2194 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1075...  Training loss: 1.6663...  6.2869 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1076...  Training loss: 1.6559...  6.2257 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1077...  Training loss: 1.6413...  6.2039 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1078...  Training loss: 1.6628...  6.2183 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1079...  Training loss: 1.6494...  6.2750 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1080...  Training loss: 1.6353...  6.2418 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1081...  Training loss: 1.6423...  6.2701 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1082...  Training loss: 1.6569...  6.2608 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1083...  Training loss: 1.6581...  6.2003 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1084...  Training loss: 1.6649...  6.2782 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1085...  Training loss: 1.6738...  6.2372 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1086...  Training loss: 1.6529...  6.2770 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1087...  Training loss: 1.6530...  6.2673 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1088...  Training loss: 1.6591...  6.2419 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1089...  Training loss: 1.6711...  6.2076 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1090...  Training loss: 1.6292...  6.2724 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1091...  Training loss: 1.6566...  6.2337 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1092...  Training loss: 1.6446...  6.1806 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1093...  Training loss: 1.6350...  6.2031 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1094...  Training loss: 1.6625...  6.2387 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1095...  Training loss: 1.6521...  6.2122 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1096...  Training loss: 1.6430...  6.2224 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1097...  Training loss: 1.6252...  6.2033 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1098...  Training loss: 1.6684...  6.2480 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1099...  Training loss: 1.6416...  6.1826 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1100...  Training loss: 1.6564...  6.2510 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1101...  Training loss: 1.7115...  6.2068 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1102...  Training loss: 1.6420...  6.1916 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1103...  Training loss: 1.6753...  6.2799 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1104...  Training loss: 1.6644...  6.2414 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1105...  Training loss: 1.6525...  6.1938 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1106...  Training loss: 1.6606...  6.2875 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1107...  Training loss: 1.6361...  6.1920 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1108...  Training loss: 1.6534...  6.2664 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1109...  Training loss: 1.6302...  6.2298 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1110...  Training loss: 1.6110...  6.2258 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1111...  Training loss: 1.6374...  6.2338 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1112...  Training loss: 1.6411...  6.2126 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1113...  Training loss: 1.6253...  6.2542 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1114...  Training loss: 1.6723...  6.2520 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1115...  Training loss: 1.6776...  6.2386 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1116...  Training loss: 1.6642...  6.2349 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1117...  Training loss: 1.6747...  6.2155 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1118...  Training loss: 1.6679...  6.2112 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1119...  Training loss: 1.6645...  6.2801 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1120...  Training loss: 1.6477...  6.2605 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1121...  Training loss: 1.6464...  6.2529 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1122...  Training loss: 1.6450...  6.2331 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1123...  Training loss: 1.6615...  6.1833 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1124...  Training loss: 1.6712...  6.2347 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1125...  Training loss: 1.6537...  6.1935 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1126...  Training loss: 1.6651...  6.1934 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1127...  Training loss: 1.6280...  6.2186 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1128...  Training loss: 1.6694...  6.2148 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1129...  Training loss: 1.6310...  6.2045 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1130...  Training loss: 1.6459...  6.1914 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1131...  Training loss: 1.6322...  6.2041 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1132...  Training loss: 1.6668...  6.1843 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1133...  Training loss: 1.6163...  6.2198 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1134...  Training loss: 1.6163...  6.2375 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1135...  Training loss: 1.6658...  6.2645 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1136...  Training loss: 1.6447...  6.2749 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1137...  Training loss: 1.6465...  6.2033 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1138...  Training loss: 1.6182...  6.2541 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1139...  Training loss: 1.6577...  6.2539 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1140...  Training loss: 1.6520...  6.2432 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1141...  Training loss: 1.6354...  6.2047 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1142...  Training loss: 1.6496...  6.1823 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1143...  Training loss: 1.6617...  6.2237 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1144...  Training loss: 1.6437...  6.1985 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1145...  Training loss: 1.6766...  6.1964 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1146...  Training loss: 1.6819...  6.2049 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1147...  Training loss: 1.6279...  6.2761 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1148...  Training loss: 1.6248...  6.2223 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1149...  Training loss: 1.6271...  6.2644 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1150...  Training loss: 1.6621...  6.2081 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1151...  Training loss: 1.6623...  6.2249 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1152...  Training loss: 1.6497...  6.2243 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1153...  Training loss: 1.6624...  6.2714 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1154...  Training loss: 1.6581...  6.2602 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1155...  Training loss: 1.6408...  6.2169 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1156...  Training loss: 1.6072...  6.2678 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1157...  Training loss: 1.6386...  6.2719 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1158...  Training loss: 1.6182...  6.2414 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1159...  Training loss: 1.6458...  6.2319 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1160...  Training loss: 1.6399...  6.2646 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1161...  Training loss: 1.6171...  6.2671 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1162...  Training loss: 1.6348...  6.2434 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1163...  Training loss: 1.6279...  6.2755 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1164...  Training loss: 1.6545...  6.1787 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1165...  Training loss: 1.6531...  6.2133 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1166...  Training loss: 1.6318...  6.2183 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1167...  Training loss: 1.6442...  6.2201 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1168...  Training loss: 1.6254...  6.1785 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1169...  Training loss: 1.6319...  6.2068 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1170...  Training loss: 1.6096...  6.2384 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1171...  Training loss: 1.6380...  6.2031 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1172...  Training loss: 1.6168...  6.1888 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1173...  Training loss: 1.6327...  6.2932 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1174...  Training loss: 1.6293...  6.2391 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1175...  Training loss: 1.6233...  6.2346 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1176...  Training loss: 1.6064...  6.2111 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1177...  Training loss: 1.6334...  6.2178 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1178...  Training loss: 1.6446...  6.3028 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1179...  Training loss: 1.6238...  6.1863 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1180...  Training loss: 1.6338...  6.1871 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1181...  Training loss: 1.6103...  6.1904 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1182...  Training loss: 1.6189...  6.2538 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1183...  Training loss: 1.6323...  6.2267 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1184...  Training loss: 1.6358...  6.1858 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1185...  Training loss: 1.6167...  6.2153 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1186...  Training loss: 1.6130...  6.2329 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1187...  Training loss: 1.5812...  6.2566 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1188...  Training loss: 1.6174...  6.2553 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1189...  Training loss: 1.6012...  6.2647 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1190...  Training loss: 1.5971...  6.2352 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1191...  Training loss: 1.6136...  6.2074 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1192...  Training loss: 1.6091...  6.2516 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1193...  Training loss: 1.6161...  6.2361 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1194...  Training loss: 1.5973...  6.2107 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1195...  Training loss: 1.6007...  6.2287 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1196...  Training loss: 1.6078...  6.3087 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1197...  Training loss: 1.5792...  6.2142 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1198...  Training loss: 1.6095...  6.2606 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1199...  Training loss: 1.6309...  6.2353 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1200...  Training loss: 1.6475...  6.2477 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1201...  Training loss: 1.6077...  6.2883 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1202...  Training loss: 1.6322...  6.2824 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1203...  Training loss: 1.6103...  6.2208 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1204...  Training loss: 1.6236...  6.2057 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1205...  Training loss: 1.6364...  6.2559 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1206...  Training loss: 1.6340...  6.2325 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1207...  Training loss: 1.6385...  6.1794 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1208...  Training loss: 1.6034...  6.1917 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1209...  Training loss: 1.6579...  6.2535 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1210...  Training loss: 1.6225...  6.2640 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1211...  Training loss: 1.5977...  6.2050 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1212...  Training loss: 1.6126...  6.2158 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1213...  Training loss: 1.5699...  6.2196 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1214...  Training loss: 1.6464...  6.2056 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1215...  Training loss: 1.6027...  6.2175 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1216...  Training loss: 1.5617...  6.2892 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1217...  Training loss: 1.6283...  6.2633 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1218...  Training loss: 1.6128...  6.1991 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1219...  Training loss: 1.6353...  6.2112 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1220...  Training loss: 1.6100...  6.2208 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1221...  Training loss: 1.5967...  6.2540 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1222...  Training loss: 1.6149...  6.2171 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1223...  Training loss: 1.5920...  6.2136 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1224...  Training loss: 1.6089...  6.2083 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1225...  Training loss: 1.6228...  6.2456 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1226...  Training loss: 1.6056...  6.2359 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1227...  Training loss: 1.6178...  6.2382 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1228...  Training loss: 1.6099...  6.2275 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1229...  Training loss: 1.6256...  6.2754 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1230...  Training loss: 1.6201...  6.2360 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1231...  Training loss: 1.5928...  6.2021 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1232...  Training loss: 1.6495...  6.3016 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1233...  Training loss: 1.6085...  6.2061 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1234...  Training loss: 1.5962...  6.1759 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1235...  Training loss: 1.5914...  6.2532 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1236...  Training loss: 1.6271...  6.2068 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1237...  Training loss: 1.6270...  6.2563 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1238...  Training loss: 1.5926...  6.2193 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1239...  Training loss: 1.5973...  6.1999 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1240...  Training loss: 1.6246...  6.2174 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1241...  Training loss: 1.6057...  6.2416 sec/batch\n",
      "Epoch: 6/20...  Training Step: 1242...  Training loss: 1.6203...  6.2341 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1243...  Training loss: 1.6905...  6.2164 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1244...  Training loss: 1.6206...  6.1803 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1245...  Training loss: 1.6378...  6.2948 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1246...  Training loss: 1.6575...  6.2039 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1247...  Training loss: 1.6121...  6.2187 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1248...  Training loss: 1.6132...  6.2306 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1249...  Training loss: 1.6198...  6.1976 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1250...  Training loss: 1.6264...  6.2506 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1251...  Training loss: 1.6324...  6.2079 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1252...  Training loss: 1.6142...  6.2306 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1253...  Training loss: 1.6083...  6.2225 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1254...  Training loss: 1.5908...  6.2102 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1255...  Training loss: 1.6247...  6.2237 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1256...  Training loss: 1.6224...  6.2220 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1257...  Training loss: 1.6047...  6.2360 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1258...  Training loss: 1.5766...  6.2563 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1259...  Training loss: 1.5738...  6.2542 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1260...  Training loss: 1.5951...  6.2067 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1261...  Training loss: 1.6082...  6.2138 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1262...  Training loss: 1.5988...  6.1977 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1263...  Training loss: 1.5878...  6.1986 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1264...  Training loss: 1.5953...  6.2212 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1265...  Training loss: 1.5693...  6.2546 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1266...  Training loss: 1.5909...  6.1855 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1267...  Training loss: 1.5741...  6.1925 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1268...  Training loss: 1.6066...  6.2637 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1269...  Training loss: 1.5943...  6.1724 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1270...  Training loss: 1.5641...  6.2152 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1271...  Training loss: 1.5720...  6.2469 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1272...  Training loss: 1.6011...  6.2159 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1273...  Training loss: 1.5659...  6.2691 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1274...  Training loss: 1.6084...  6.2361 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1275...  Training loss: 1.6186...  6.2372 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1276...  Training loss: 1.5995...  6.2111 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1277...  Training loss: 1.5928...  6.2730 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1278...  Training loss: 1.5805...  6.2070 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1279...  Training loss: 1.5880...  6.2352 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1280...  Training loss: 1.5706...  6.2129 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1281...  Training loss: 1.5727...  6.2197 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1282...  Training loss: 1.5791...  6.2378 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1283...  Training loss: 1.5840...  6.2621 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1284...  Training loss: 1.5674...  6.1716 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1285...  Training loss: 1.5821...  6.1969 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1286...  Training loss: 1.5745...  6.1968 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1287...  Training loss: 1.5645...  6.1710 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1288...  Training loss: 1.5669...  6.2434 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1289...  Training loss: 1.5808...  6.2460 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1290...  Training loss: 1.5908...  6.2487 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1291...  Training loss: 1.5918...  6.2263 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1292...  Training loss: 1.6013...  6.2241 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1293...  Training loss: 1.5710...  6.2116 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1294...  Training loss: 1.5642...  6.1992 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1295...  Training loss: 1.5742...  6.2831 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1296...  Training loss: 1.5778...  6.2443 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1297...  Training loss: 1.5452...  6.2044 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1298...  Training loss: 1.5773...  6.2526 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1299...  Training loss: 1.5646...  6.2958 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1300...  Training loss: 1.5636...  6.1933 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1301...  Training loss: 1.5861...  6.2234 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1302...  Training loss: 1.5833...  6.2840 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1303...  Training loss: 1.5676...  6.2326 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1304...  Training loss: 1.5500...  6.2149 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1305...  Training loss: 1.5806...  6.2571 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1306...  Training loss: 1.5743...  6.2645 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1307...  Training loss: 1.5827...  6.1717 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1308...  Training loss: 1.6309...  6.2619 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1309...  Training loss: 1.5673...  6.2243 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1310...  Training loss: 1.6042...  6.2468 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1311...  Training loss: 1.5962...  6.2035 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1312...  Training loss: 1.5830...  6.2186 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1313...  Training loss: 1.6005...  6.2678 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1314...  Training loss: 1.5523...  6.2356 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1315...  Training loss: 1.5785...  6.2016 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1316...  Training loss: 1.5655...  6.2235 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1317...  Training loss: 1.5464...  6.2323 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1318...  Training loss: 1.5771...  6.1862 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1319...  Training loss: 1.5599...  6.1911 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1320...  Training loss: 1.5377...  6.2114 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1321...  Training loss: 1.5867...  6.2273 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1322...  Training loss: 1.5728...  6.2166 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1323...  Training loss: 1.5633...  6.2475 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1324...  Training loss: 1.5729...  6.2508 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1325...  Training loss: 1.5705...  6.2559 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1326...  Training loss: 1.5786...  6.1949 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1327...  Training loss: 1.5580...  6.2354 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1328...  Training loss: 1.5537...  6.2677 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1329...  Training loss: 1.5643...  6.2005 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1330...  Training loss: 1.5682...  6.2541 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1331...  Training loss: 1.5781...  6.2213 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1332...  Training loss: 1.5642...  6.2158 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1333...  Training loss: 1.5650...  6.2307 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1334...  Training loss: 1.5360...  6.2259 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1335...  Training loss: 1.5793...  6.2453 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1336...  Training loss: 1.5382...  6.2524 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1337...  Training loss: 1.5572...  6.2516 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1338...  Training loss: 1.5490...  6.2018 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1339...  Training loss: 1.5750...  6.1980 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1340...  Training loss: 1.5348...  6.2755 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1341...  Training loss: 1.5189...  6.2737 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1342...  Training loss: 1.5897...  6.2286 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1343...  Training loss: 1.5704...  6.2479 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1344...  Training loss: 1.5631...  6.2471 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1345...  Training loss: 1.5378...  6.2085 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1346...  Training loss: 1.5764...  6.1738 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1347...  Training loss: 1.5709...  6.1996 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1348...  Training loss: 1.5513...  6.2151 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1349...  Training loss: 1.5697...  6.2063 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1350...  Training loss: 1.5777...  6.1962 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1351...  Training loss: 1.5614...  6.2129 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1352...  Training loss: 1.6059...  6.2331 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1353...  Training loss: 1.6106...  6.2268 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1354...  Training loss: 1.5513...  6.1964 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1355...  Training loss: 1.5358...  6.1842 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1356...  Training loss: 1.5434...  6.2413 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1357...  Training loss: 1.5765...  6.2537 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1358...  Training loss: 1.5866...  6.2019 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1359...  Training loss: 1.5807...  6.2259 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1360...  Training loss: 1.5789...  6.2104 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1361...  Training loss: 1.5786...  6.2500 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1362...  Training loss: 1.5551...  6.1722 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1363...  Training loss: 1.5304...  6.1781 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1364...  Training loss: 1.5574...  6.2274 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1365...  Training loss: 1.5431...  6.2410 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1366...  Training loss: 1.5626...  6.2644 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1367...  Training loss: 1.5645...  6.2711 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1368...  Training loss: 1.5360...  6.2399 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1369...  Training loss: 1.5485...  6.2298 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1370...  Training loss: 1.5510...  6.2009 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1371...  Training loss: 1.5769...  6.2190 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1372...  Training loss: 1.5863...  6.3105 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1373...  Training loss: 1.5629...  6.2252 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1374...  Training loss: 1.5643...  6.2730 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1375...  Training loss: 1.5482...  6.2417 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1376...  Training loss: 1.5500...  6.1991 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1377...  Training loss: 1.5338...  6.2013 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1378...  Training loss: 1.5545...  6.1914 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1379...  Training loss: 1.5431...  6.2282 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1380...  Training loss: 1.5596...  6.2545 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1381...  Training loss: 1.5480...  6.2411 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1382...  Training loss: 1.5404...  6.2134 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1383...  Training loss: 1.5308...  6.2354 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1384...  Training loss: 1.5492...  6.2192 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1385...  Training loss: 1.5633...  6.1815 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1386...  Training loss: 1.5388...  6.2537 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1387...  Training loss: 1.5622...  6.2183 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1388...  Training loss: 1.5352...  6.2363 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1389...  Training loss: 1.5328...  6.2276 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1390...  Training loss: 1.5499...  6.2264 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1391...  Training loss: 1.5535...  6.2951 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1392...  Training loss: 1.5243...  6.2512 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1393...  Training loss: 1.5441...  6.2656 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1394...  Training loss: 1.5166...  6.2043 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1395...  Training loss: 1.5413...  6.2221 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1396...  Training loss: 1.5231...  6.2300 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1397...  Training loss: 1.5188...  6.2284 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1398...  Training loss: 1.5389...  6.1796 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1399...  Training loss: 1.5334...  6.2133 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1400...  Training loss: 1.5306...  6.2057 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1401...  Training loss: 1.5266...  6.2793 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1402...  Training loss: 1.5184...  6.1995 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1403...  Training loss: 1.5310...  6.2355 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1404...  Training loss: 1.4972...  6.2216 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1405...  Training loss: 1.5365...  6.1977 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1406...  Training loss: 1.5402...  6.2090 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1407...  Training loss: 1.5771...  6.2227 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1408...  Training loss: 1.5374...  6.2231 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1409...  Training loss: 1.5657...  6.2324 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1410...  Training loss: 1.5287...  6.2541 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1411...  Training loss: 1.5531...  6.2336 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1412...  Training loss: 1.5553...  6.2502 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1413...  Training loss: 1.5590...  6.1761 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1414...  Training loss: 1.5731...  6.2085 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1415...  Training loss: 1.5384...  6.2194 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1416...  Training loss: 1.5946...  6.2713 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1417...  Training loss: 1.5518...  6.2494 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1418...  Training loss: 1.5312...  6.2584 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1419...  Training loss: 1.5524...  6.1604 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1420...  Training loss: 1.5181...  6.2133 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1421...  Training loss: 1.5825...  6.2342 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1422...  Training loss: 1.5377...  6.2501 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1423...  Training loss: 1.5049...  6.2512 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1424...  Training loss: 1.5533...  6.2689 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1425...  Training loss: 1.5517...  6.2018 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1426...  Training loss: 1.5694...  6.1896 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1427...  Training loss: 1.5488...  6.2480 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1428...  Training loss: 1.5392...  6.1759 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1429...  Training loss: 1.5493...  6.2108 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1430...  Training loss: 1.5263...  6.2469 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1431...  Training loss: 1.5357...  6.1966 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1432...  Training loss: 1.5656...  6.2261 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1433...  Training loss: 1.5437...  6.1935 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1434...  Training loss: 1.5506...  6.2020 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1435...  Training loss: 1.5486...  6.2004 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1436...  Training loss: 1.5548...  6.2418 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1437...  Training loss: 1.5427...  6.2455 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1438...  Training loss: 1.5294...  6.2308 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1439...  Training loss: 1.5869...  6.2164 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1440...  Training loss: 1.5429...  6.2309 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1441...  Training loss: 1.5266...  6.2027 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1442...  Training loss: 1.5219...  6.1754 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1443...  Training loss: 1.5741...  6.2150 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1444...  Training loss: 1.5541...  6.1889 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1445...  Training loss: 1.5183...  6.1457 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1446...  Training loss: 1.5264...  6.1963 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1447...  Training loss: 1.5628...  6.1930 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1448...  Training loss: 1.5528...  6.2510 sec/batch\n",
      "Epoch: 7/20...  Training Step: 1449...  Training loss: 1.5569...  6.2201 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1450...  Training loss: 1.6522...  6.2024 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1451...  Training loss: 1.5702...  6.2069 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1452...  Training loss: 1.6008...  6.2434 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1453...  Training loss: 1.6424...  6.2029 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1454...  Training loss: 1.5880...  6.1866 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1455...  Training loss: 1.5757...  6.2407 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1456...  Training loss: 1.5820...  6.2635 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1457...  Training loss: 1.5886...  6.3016 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1458...  Training loss: 1.5982...  6.3472 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1459...  Training loss: 1.5907...  6.2797 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1460...  Training loss: 1.5742...  6.2864 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1461...  Training loss: 1.5457...  6.2975 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1462...  Training loss: 1.5981...  6.3187 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1463...  Training loss: 1.5795...  6.3094 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1464...  Training loss: 1.5766...  6.2401 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1465...  Training loss: 1.5388...  6.2255 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1466...  Training loss: 1.5394...  6.2498 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1467...  Training loss: 1.5537...  6.2299 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1468...  Training loss: 1.5767...  6.2493 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1469...  Training loss: 1.5673...  6.2496 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1470...  Training loss: 1.5495...  6.1988 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1471...  Training loss: 1.5596...  6.1979 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1472...  Training loss: 1.5367...  6.1922 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1473...  Training loss: 1.5534...  6.1979 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1474...  Training loss: 1.5499...  6.2093 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1475...  Training loss: 1.5750...  6.1578 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1476...  Training loss: 1.5650...  6.2860 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1477...  Training loss: 1.5396...  6.2418 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1478...  Training loss: 1.5414...  6.2363 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1479...  Training loss: 1.5649...  6.2170 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1480...  Training loss: 1.5284...  6.2628 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1481...  Training loss: 1.5691...  6.3477 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1482...  Training loss: 1.5716...  6.2821 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1483...  Training loss: 1.5593...  6.1886 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1484...  Training loss: 1.5643...  6.1870 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1485...  Training loss: 1.5374...  6.2267 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1486...  Training loss: 1.5657...  6.2210 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1487...  Training loss: 1.5395...  6.2023 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1488...  Training loss: 1.5347...  6.1985 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1489...  Training loss: 1.5651...  6.1837 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1490...  Training loss: 1.5407...  6.2214 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1491...  Training loss: 1.5230...  6.1902 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1492...  Training loss: 1.5415...  6.2336 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1493...  Training loss: 1.5280...  6.1907 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1494...  Training loss: 1.5235...  6.2329 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1495...  Training loss: 1.5373...  6.3134 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1496...  Training loss: 1.5466...  6.2717 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1497...  Training loss: 1.5523...  6.1652 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1498...  Training loss: 1.5402...  6.2808 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1499...  Training loss: 1.5630...  6.2832 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1500...  Training loss: 1.5349...  6.1861 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1501...  Training loss: 1.5313...  6.1950 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1502...  Training loss: 1.5273...  6.2047 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1503...  Training loss: 1.5324...  6.1917 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1504...  Training loss: 1.5136...  6.1972 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1505...  Training loss: 1.5488...  6.2141 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1506...  Training loss: 1.5321...  6.2011 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1507...  Training loss: 1.5178...  6.2461 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1508...  Training loss: 1.5606...  6.2183 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1509...  Training loss: 1.5475...  6.2088 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1510...  Training loss: 1.5283...  6.1966 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1511...  Training loss: 1.5183...  6.2132 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1512...  Training loss: 1.5464...  6.1971 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1513...  Training loss: 1.5281...  6.2275 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1514...  Training loss: 1.5581...  6.2634 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1515...  Training loss: 1.5987...  6.1885 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1516...  Training loss: 1.5433...  6.1876 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1517...  Training loss: 1.5640...  6.2501 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1518...  Training loss: 1.5542...  6.2686 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1519...  Training loss: 1.5395...  6.2066 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1520...  Training loss: 1.5478...  6.2155 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1521...  Training loss: 1.5144...  6.2115 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1522...  Training loss: 1.5250...  6.1968 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1523...  Training loss: 1.5180...  6.2390 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1524...  Training loss: 1.5058...  6.2195 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1525...  Training loss: 1.5258...  6.1838 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1526...  Training loss: 1.5115...  6.1960 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1527...  Training loss: 1.4863...  6.1855 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1528...  Training loss: 1.5355...  6.1993 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1529...  Training loss: 1.5221...  6.2365 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1530...  Training loss: 1.5204...  6.2382 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1531...  Training loss: 1.5271...  6.2509 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1532...  Training loss: 1.5185...  6.2770 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1533...  Training loss: 1.5286...  6.2218 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1534...  Training loss: 1.5181...  6.2771 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1535...  Training loss: 1.5098...  6.2109 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1536...  Training loss: 1.5159...  6.2062 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1537...  Training loss: 1.5263...  6.2750 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1538...  Training loss: 1.5354...  6.2378 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1539...  Training loss: 1.5097...  6.2325 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1540...  Training loss: 1.5117...  6.2822 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1541...  Training loss: 1.4956...  6.2179 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1542...  Training loss: 1.5270...  6.2027 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1543...  Training loss: 1.4969...  6.2170 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1544...  Training loss: 1.5096...  6.2024 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1545...  Training loss: 1.5007...  6.1762 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1546...  Training loss: 1.5257...  6.1754 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1547...  Training loss: 1.4845...  6.2040 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1548...  Training loss: 1.4815...  6.2291 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1549...  Training loss: 1.5413...  6.2256 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1550...  Training loss: 1.5183...  6.2478 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1551...  Training loss: 1.5280...  6.2295 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1552...  Training loss: 1.4898...  6.2246 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1553...  Training loss: 1.5276...  6.2200 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1554...  Training loss: 1.5212...  6.2332 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1555...  Training loss: 1.5091...  6.2047 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1556...  Training loss: 1.5231...  6.2004 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1557...  Training loss: 1.5254...  6.2148 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1558...  Training loss: 1.5174...  6.2047 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1559...  Training loss: 1.5463...  6.1924 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1560...  Training loss: 1.5496...  6.1932 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1561...  Training loss: 1.5009...  6.1831 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1562...  Training loss: 1.4883...  6.2470 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1563...  Training loss: 1.4961...  6.2211 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1564...  Training loss: 1.5362...  6.2357 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1565...  Training loss: 1.5501...  6.2052 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1566...  Training loss: 1.5307...  6.2030 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1567...  Training loss: 1.5342...  6.2153 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1568...  Training loss: 1.5252...  6.2491 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1569...  Training loss: 1.5055...  6.2286 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1570...  Training loss: 1.4695...  6.2140 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1571...  Training loss: 1.5124...  6.1849 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1572...  Training loss: 1.4930...  6.2057 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1573...  Training loss: 1.5158...  6.2499 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1574...  Training loss: 1.5185...  6.2168 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1575...  Training loss: 1.4867...  6.2010 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1576...  Training loss: 1.4929...  6.2477 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1577...  Training loss: 1.5121...  6.2386 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1578...  Training loss: 1.5171...  6.2520 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1579...  Training loss: 1.5356...  6.2147 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1580...  Training loss: 1.5035...  6.1796 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1581...  Training loss: 1.5119...  6.2342 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1582...  Training loss: 1.5015...  6.2637 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1583...  Training loss: 1.5030...  6.2160 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1584...  Training loss: 1.4866...  6.2484 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1585...  Training loss: 1.5122...  6.1952 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1586...  Training loss: 1.4953...  6.2263 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1587...  Training loss: 1.5099...  6.1858 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1588...  Training loss: 1.5078...  6.2398 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1589...  Training loss: 1.4986...  6.2613 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1590...  Training loss: 1.4869...  6.2139 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1591...  Training loss: 1.5033...  6.2917 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1592...  Training loss: 1.5235...  6.2801 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1593...  Training loss: 1.4883...  6.1608 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1594...  Training loss: 1.5001...  6.2330 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1595...  Training loss: 1.4921...  6.1784 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1596...  Training loss: 1.4959...  6.1760 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1597...  Training loss: 1.5029...  6.2323 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1598...  Training loss: 1.5048...  6.1773 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1599...  Training loss: 1.4851...  6.2044 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1600...  Training loss: 1.4866...  6.2456 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1601...  Training loss: 1.4642...  6.2658 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1602...  Training loss: 1.4944...  6.1984 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1603...  Training loss: 1.4836...  6.2139 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1604...  Training loss: 1.4732...  6.1800 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1605...  Training loss: 1.4945...  6.2295 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1606...  Training loss: 1.4726...  6.2266 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1607...  Training loss: 1.4860...  6.2568 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1608...  Training loss: 1.4792...  6.1744 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1609...  Training loss: 1.4637...  6.2088 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1610...  Training loss: 1.4770...  6.2254 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1611...  Training loss: 1.4501...  6.1923 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1612...  Training loss: 1.5023...  6.1793 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1613...  Training loss: 1.5023...  6.2098 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1614...  Training loss: 1.5233...  6.1883 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1615...  Training loss: 1.4961...  6.1580 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1616...  Training loss: 1.5276...  6.1486 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1617...  Training loss: 1.4841...  6.2235 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1618...  Training loss: 1.5211...  6.2063 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1619...  Training loss: 1.5140...  6.2699 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1620...  Training loss: 1.5184...  6.2389 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1621...  Training loss: 1.5261...  6.2533 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1622...  Training loss: 1.4882...  6.2296 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1623...  Training loss: 1.5468...  6.2005 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1624...  Training loss: 1.5258...  6.2030 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1625...  Training loss: 1.4896...  6.2129 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1626...  Training loss: 1.5118...  6.2378 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1627...  Training loss: 1.4685...  6.2351 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1628...  Training loss: 1.5311...  6.2351 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1629...  Training loss: 1.4929...  6.2426 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1630...  Training loss: 1.4586...  6.2118 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1631...  Training loss: 1.5135...  6.2357 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1632...  Training loss: 1.5094...  6.2153 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1633...  Training loss: 1.5242...  6.1681 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1634...  Training loss: 1.5014...  6.1734 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1635...  Training loss: 1.4907...  6.2703 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1636...  Training loss: 1.5034...  6.2502 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1637...  Training loss: 1.4818...  6.1815 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1638...  Training loss: 1.4854...  6.2220 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1639...  Training loss: 1.5293...  6.2397 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1640...  Training loss: 1.5014...  6.2407 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1641...  Training loss: 1.5039...  6.2314 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1642...  Training loss: 1.5050...  6.1812 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1643...  Training loss: 1.4988...  6.2168 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1644...  Training loss: 1.5025...  6.1962 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1645...  Training loss: 1.4840...  6.2770 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1646...  Training loss: 1.5358...  6.2341 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1647...  Training loss: 1.4975...  6.2473 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1648...  Training loss: 1.4897...  6.2027 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1649...  Training loss: 1.4918...  6.2248 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1650...  Training loss: 1.5275...  6.1962 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1651...  Training loss: 1.5239...  6.2701 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1652...  Training loss: 1.4859...  6.2418 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1653...  Training loss: 1.4990...  6.2778 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1654...  Training loss: 1.5323...  6.1870 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1655...  Training loss: 1.5122...  6.2340 sec/batch\n",
      "Epoch: 8/20...  Training Step: 1656...  Training loss: 1.5140...  6.1950 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1657...  Training loss: 1.5847...  6.1893 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1658...  Training loss: 1.5034...  6.2270 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1659...  Training loss: 1.5147...  6.3032 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1660...  Training loss: 1.5342...  6.2398 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1661...  Training loss: 1.4984...  6.1917 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1662...  Training loss: 1.4950...  6.2081 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1663...  Training loss: 1.5114...  6.2009 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1664...  Training loss: 1.5150...  6.2388 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1665...  Training loss: 1.5204...  6.1798 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1666...  Training loss: 1.5073...  6.2321 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1667...  Training loss: 1.5078...  6.2516 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1668...  Training loss: 1.4652...  6.2170 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1669...  Training loss: 1.5156...  6.2928 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1670...  Training loss: 1.5091...  6.1930 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1671...  Training loss: 1.5003...  6.2389 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1672...  Training loss: 1.4685...  6.2594 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1673...  Training loss: 1.4705...  6.1824 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1674...  Training loss: 1.4852...  6.2274 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1675...  Training loss: 1.4945...  6.2094 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1676...  Training loss: 1.5043...  6.2318 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1677...  Training loss: 1.4814...  6.2010 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1678...  Training loss: 1.4910...  6.2235 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1679...  Training loss: 1.4830...  6.2159 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1680...  Training loss: 1.4903...  6.2099 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1681...  Training loss: 1.4786...  6.2316 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1682...  Training loss: 1.5122...  6.1615 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1683...  Training loss: 1.4879...  6.2137 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1684...  Training loss: 1.4724...  6.2360 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1685...  Training loss: 1.4716...  6.2191 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1686...  Training loss: 1.4950...  6.2049 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1687...  Training loss: 1.4634...  6.2765 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1688...  Training loss: 1.5006...  6.1943 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1689...  Training loss: 1.5197...  6.2198 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1690...  Training loss: 1.4926...  6.1804 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1691...  Training loss: 1.4951...  6.2082 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1692...  Training loss: 1.4698...  6.2431 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1693...  Training loss: 1.4979...  6.1554 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1694...  Training loss: 1.4692...  6.2555 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1695...  Training loss: 1.4679...  6.2153 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1696...  Training loss: 1.4886...  6.2393 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1697...  Training loss: 1.4759...  6.2566 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1698...  Training loss: 1.4664...  6.2543 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1699...  Training loss: 1.4903...  6.2079 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1700...  Training loss: 1.4661...  6.2334 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1701...  Training loss: 1.4671...  6.2341 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1702...  Training loss: 1.4842...  6.1911 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1703...  Training loss: 1.4903...  6.2553 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1704...  Training loss: 1.4877...  6.2159 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1705...  Training loss: 1.4781...  6.1840 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1706...  Training loss: 1.5063...  6.2637 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1707...  Training loss: 1.4822...  6.2323 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1708...  Training loss: 1.4771...  6.2044 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1709...  Training loss: 1.4737...  6.2343 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1710...  Training loss: 1.4775...  6.2369 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1711...  Training loss: 1.4514...  6.2571 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1712...  Training loss: 1.4869...  6.2424 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1713...  Training loss: 1.4639...  6.1923 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1714...  Training loss: 1.4663...  6.2162 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1715...  Training loss: 1.4922...  6.2054 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1716...  Training loss: 1.4836...  6.2159 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1717...  Training loss: 1.4641...  6.2057 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1718...  Training loss: 1.4600...  6.2038 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1719...  Training loss: 1.4903...  6.1814 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1720...  Training loss: 1.4689...  6.2602 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1721...  Training loss: 1.4961...  6.1995 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1722...  Training loss: 1.5377...  6.2682 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1723...  Training loss: 1.4759...  6.2078 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1724...  Training loss: 1.5046...  6.2161 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1725...  Training loss: 1.4901...  6.1717 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1726...  Training loss: 1.4842...  6.2809 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1727...  Training loss: 1.4954...  6.2364 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1728...  Training loss: 1.4670...  6.2061 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1729...  Training loss: 1.4699...  6.2289 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1730...  Training loss: 1.4679...  6.2276 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1731...  Training loss: 1.4509...  6.2196 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1732...  Training loss: 1.4727...  6.1837 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1733...  Training loss: 1.4546...  6.2097 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1734...  Training loss: 1.4368...  6.2714 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1735...  Training loss: 1.4805...  6.2135 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1736...  Training loss: 1.4760...  6.1874 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1737...  Training loss: 1.4780...  6.1803 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1738...  Training loss: 1.4778...  6.2050 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1739...  Training loss: 1.4862...  6.2628 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1740...  Training loss: 1.4920...  6.2135 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1741...  Training loss: 1.4614...  6.2384 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1742...  Training loss: 1.4580...  6.2332 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1743...  Training loss: 1.4668...  6.1988 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1744...  Training loss: 1.4745...  6.1825 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1745...  Training loss: 1.4748...  6.2489 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1746...  Training loss: 1.4722...  6.2040 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1747...  Training loss: 1.4635...  6.2082 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1748...  Training loss: 1.4471...  6.2595 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1749...  Training loss: 1.4881...  6.2336 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1750...  Training loss: 1.4515...  6.2530 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1751...  Training loss: 1.4678...  6.2747 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1752...  Training loss: 1.4544...  6.1396 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1753...  Training loss: 1.4873...  6.2373 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1754...  Training loss: 1.4399...  6.1931 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1755...  Training loss: 1.4349...  6.2566 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1756...  Training loss: 1.4861...  6.2535 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1757...  Training loss: 1.4720...  6.2302 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1758...  Training loss: 1.4773...  6.1764 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1759...  Training loss: 1.4360...  6.2331 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1760...  Training loss: 1.4728...  6.2142 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1761...  Training loss: 1.4767...  6.1946 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1762...  Training loss: 1.4665...  6.2286 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1763...  Training loss: 1.4842...  6.2180 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1764...  Training loss: 1.4873...  6.2193 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1765...  Training loss: 1.4833...  6.2049 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1766...  Training loss: 1.5019...  6.2095 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1767...  Training loss: 1.5016...  6.2248 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1768...  Training loss: 1.4608...  6.1964 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1769...  Training loss: 1.4467...  6.2278 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1770...  Training loss: 1.4634...  6.1829 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1771...  Training loss: 1.5005...  6.2124 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1772...  Training loss: 1.4957...  6.2482 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1773...  Training loss: 1.4881...  6.1687 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1774...  Training loss: 1.4953...  6.2606 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1775...  Training loss: 1.4804...  6.1815 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1776...  Training loss: 1.4641...  6.2254 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1777...  Training loss: 1.4461...  6.1945 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1778...  Training loss: 1.4611...  6.1892 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1779...  Training loss: 1.4544...  6.2486 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1780...  Training loss: 1.4672...  6.1724 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1781...  Training loss: 1.4802...  6.2505 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1782...  Training loss: 1.4511...  6.2175 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1783...  Training loss: 1.4717...  6.2445 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1784...  Training loss: 1.4572...  6.2298 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1785...  Training loss: 1.4797...  6.2187 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1786...  Training loss: 1.4961...  6.2430 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1787...  Training loss: 1.4686...  6.2051 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1788...  Training loss: 1.4702...  6.2041 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1789...  Training loss: 1.4750...  6.2206 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1790...  Training loss: 1.4610...  6.2577 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1791...  Training loss: 1.4454...  6.2350 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1792...  Training loss: 1.4723...  6.2269 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1793...  Training loss: 1.4532...  6.2384 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1794...  Training loss: 1.4699...  6.1681 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1795...  Training loss: 1.4579...  6.2421 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1796...  Training loss: 1.4512...  6.2312 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1797...  Training loss: 1.4441...  6.1895 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1798...  Training loss: 1.4659...  6.1966 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1799...  Training loss: 1.4845...  6.2017 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1800...  Training loss: 1.4539...  6.2170 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1801...  Training loss: 1.4659...  6.2143 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1802...  Training loss: 1.4522...  6.1755 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1803...  Training loss: 1.4519...  6.2817 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1804...  Training loss: 1.4611...  6.2107 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1805...  Training loss: 1.4701...  6.2700 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1806...  Training loss: 1.4462...  6.1929 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1807...  Training loss: 1.4420...  6.1925 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1808...  Training loss: 1.4277...  6.1997 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1809...  Training loss: 1.4468...  6.2595 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1810...  Training loss: 1.4440...  6.2098 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1811...  Training loss: 1.4324...  6.2814 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1812...  Training loss: 1.4541...  6.2275 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1813...  Training loss: 1.4300...  6.2050 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1814...  Training loss: 1.4442...  6.2567 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1815...  Training loss: 1.4371...  6.2245 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1816...  Training loss: 1.4135...  6.2064 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1817...  Training loss: 1.4268...  6.2231 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1818...  Training loss: 1.4104...  6.2031 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1819...  Training loss: 1.4621...  6.1733 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1820...  Training loss: 1.4508...  6.2354 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1821...  Training loss: 1.4959...  6.1875 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1822...  Training loss: 1.4542...  6.2775 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1823...  Training loss: 1.4832...  6.2433 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1824...  Training loss: 1.4499...  6.1700 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1825...  Training loss: 1.4699...  6.2346 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1826...  Training loss: 1.4849...  6.2052 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1827...  Training loss: 1.4722...  6.1629 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1828...  Training loss: 1.4857...  6.2298 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1829...  Training loss: 1.4398...  6.2036 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1830...  Training loss: 1.5080...  6.2165 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1831...  Training loss: 1.4820...  6.2120 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1832...  Training loss: 1.4469...  6.2601 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1833...  Training loss: 1.4782...  6.2354 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1834...  Training loss: 1.4400...  6.2160 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1835...  Training loss: 1.4948...  6.2234 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1836...  Training loss: 1.4558...  6.1971 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1837...  Training loss: 1.4359...  6.2079 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1838...  Training loss: 1.4815...  6.2083 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1839...  Training loss: 1.4657...  6.2338 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1840...  Training loss: 1.4846...  6.1942 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1841...  Training loss: 1.4645...  6.1802 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1842...  Training loss: 1.4577...  6.1732 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1843...  Training loss: 1.4785...  6.1800 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1844...  Training loss: 1.4530...  6.1593 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1845...  Training loss: 1.4533...  6.2079 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1846...  Training loss: 1.4830...  6.1881 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1847...  Training loss: 1.4720...  6.2349 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1848...  Training loss: 1.4666...  6.1589 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1849...  Training loss: 1.4648...  6.2689 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1850...  Training loss: 1.4738...  6.2723 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1851...  Training loss: 1.4713...  6.2748 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1852...  Training loss: 1.4511...  6.1982 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1853...  Training loss: 1.5058...  6.1890 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1854...  Training loss: 1.4598...  6.1894 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1855...  Training loss: 1.4477...  6.2106 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1856...  Training loss: 1.4445...  6.2224 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1857...  Training loss: 1.4933...  6.2019 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1858...  Training loss: 1.4821...  6.1679 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1859...  Training loss: 1.4396...  6.2416 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1860...  Training loss: 1.4482...  6.1929 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1861...  Training loss: 1.4883...  6.1805 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1862...  Training loss: 1.4673...  6.2325 sec/batch\n",
      "Epoch: 9/20...  Training Step: 1863...  Training loss: 1.4804...  6.2579 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1864...  Training loss: 1.5469...  6.1824 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1865...  Training loss: 1.4654...  6.2113 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1866...  Training loss: 1.4615...  6.2411 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1867...  Training loss: 1.5113...  6.2484 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1868...  Training loss: 1.4719...  6.2423 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1869...  Training loss: 1.4646...  6.2283 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1870...  Training loss: 1.4709...  6.2218 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1871...  Training loss: 1.4763...  6.1833 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1872...  Training loss: 1.4811...  6.2552 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1873...  Training loss: 1.4676...  6.2316 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1874...  Training loss: 1.4537...  6.1810 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1875...  Training loss: 1.4335...  6.2102 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1876...  Training loss: 1.4826...  6.1817 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1877...  Training loss: 1.4672...  6.1834 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1878...  Training loss: 1.4587...  6.2375 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1879...  Training loss: 1.4334...  6.2478 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1880...  Training loss: 1.4318...  6.2938 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1881...  Training loss: 1.4568...  6.3537 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1882...  Training loss: 1.4606...  6.2064 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1883...  Training loss: 1.4656...  6.2145 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1884...  Training loss: 1.4468...  6.2033 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1885...  Training loss: 1.4527...  6.2265 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1886...  Training loss: 1.4450...  6.2110 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1887...  Training loss: 1.4464...  6.2267 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1888...  Training loss: 1.4423...  6.2213 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1889...  Training loss: 1.4707...  6.2110 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1890...  Training loss: 1.4623...  6.2687 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1891...  Training loss: 1.4453...  6.2159 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1892...  Training loss: 1.4411...  6.2290 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1893...  Training loss: 1.4559...  6.2375 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1894...  Training loss: 1.4269...  6.1971 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1895...  Training loss: 1.4544...  6.2402 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1896...  Training loss: 1.4762...  6.2040 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1897...  Training loss: 1.4587...  6.2198 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1898...  Training loss: 1.4620...  6.2376 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1899...  Training loss: 1.4417...  6.2617 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1900...  Training loss: 1.4566...  6.1898 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1901...  Training loss: 1.4449...  6.2025 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1902...  Training loss: 1.4354...  6.1951 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1903...  Training loss: 1.4579...  6.1811 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1904...  Training loss: 1.4516...  6.1410 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1905...  Training loss: 1.4342...  6.2425 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1906...  Training loss: 1.4408...  6.2325 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1907...  Training loss: 1.4273...  6.2357 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1908...  Training loss: 1.4243...  6.2106 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1909...  Training loss: 1.4421...  6.2334 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1910...  Training loss: 1.4457...  6.2070 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1911...  Training loss: 1.4563...  6.1708 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1912...  Training loss: 1.4585...  6.1974 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1913...  Training loss: 1.4698...  6.2137 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1914...  Training loss: 1.4428...  6.2384 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1915...  Training loss: 1.4344...  6.2075 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1916...  Training loss: 1.4334...  6.2466 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1917...  Training loss: 1.4469...  6.1918 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1918...  Training loss: 1.4071...  6.2360 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1919...  Training loss: 1.4504...  6.2071 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1920...  Training loss: 1.4414...  6.2145 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1921...  Training loss: 1.4335...  6.2504 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1922...  Training loss: 1.4532...  6.2318 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1923...  Training loss: 1.4591...  6.2534 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1924...  Training loss: 1.4328...  6.1518 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1925...  Training loss: 1.4294...  6.2452 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1926...  Training loss: 1.4595...  6.1689 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1927...  Training loss: 1.4485...  6.2264 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1928...  Training loss: 1.4692...  6.1758 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1929...  Training loss: 1.5071...  6.2250 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1930...  Training loss: 1.4485...  6.2494 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1931...  Training loss: 1.4703...  6.2858 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1932...  Training loss: 1.4650...  6.2516 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1933...  Training loss: 1.4467...  6.2003 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1934...  Training loss: 1.4612...  6.2462 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1935...  Training loss: 1.4161...  6.1885 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1936...  Training loss: 1.4437...  6.2423 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1937...  Training loss: 1.4306...  6.2287 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1938...  Training loss: 1.4191...  6.2502 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1939...  Training loss: 1.4432...  6.2587 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1940...  Training loss: 1.4237...  6.1955 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1941...  Training loss: 1.3977...  6.1943 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1942...  Training loss: 1.4484...  6.2808 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1943...  Training loss: 1.4306...  6.2464 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1944...  Training loss: 1.4319...  6.2371 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1945...  Training loss: 1.4472...  6.2149 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1946...  Training loss: 1.4503...  6.2202 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1947...  Training loss: 1.4529...  6.1555 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1948...  Training loss: 1.4296...  6.2103 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1949...  Training loss: 1.4404...  6.2308 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1950...  Training loss: 1.4506...  6.1945 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1951...  Training loss: 1.4525...  6.1875 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1952...  Training loss: 1.4471...  6.2215 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1953...  Training loss: 1.4440...  6.2408 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1954...  Training loss: 1.4371...  6.1969 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1955...  Training loss: 1.4144...  6.2057 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1956...  Training loss: 1.4516...  6.2100 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1957...  Training loss: 1.4110...  6.2328 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1958...  Training loss: 1.4390...  6.1940 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1959...  Training loss: 1.4213...  6.1652 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1960...  Training loss: 1.4509...  6.1673 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1961...  Training loss: 1.4057...  6.2153 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1962...  Training loss: 1.4034...  6.1851 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1963...  Training loss: 1.4471...  6.2525 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1964...  Training loss: 1.4550...  6.2022 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1965...  Training loss: 1.4407...  6.1833 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1966...  Training loss: 1.4028...  6.2392 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1967...  Training loss: 1.4495...  6.2663 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1968...  Training loss: 1.4411...  6.3257 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1969...  Training loss: 1.4246...  6.3257 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1970...  Training loss: 1.4431...  6.2310 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1971...  Training loss: 1.4529...  6.2493 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1972...  Training loss: 1.4394...  6.2099 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1973...  Training loss: 1.4760...  6.2631 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1974...  Training loss: 1.4755...  6.2487 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1975...  Training loss: 1.4273...  6.2355 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1976...  Training loss: 1.4176...  6.2802 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1977...  Training loss: 1.4179...  6.3405 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1978...  Training loss: 1.4490...  6.3049 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1979...  Training loss: 1.4578...  6.1888 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1980...  Training loss: 1.4529...  6.1868 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1981...  Training loss: 1.4516...  6.2244 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1982...  Training loss: 1.4530...  6.1977 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1983...  Training loss: 1.4272...  6.1919 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1984...  Training loss: 1.4020...  6.2091 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1985...  Training loss: 1.4402...  6.2059 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1986...  Training loss: 1.4084...  6.2472 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1987...  Training loss: 1.4340...  6.1918 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1988...  Training loss: 1.4435...  6.2244 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1989...  Training loss: 1.4015...  6.2181 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1990...  Training loss: 1.4271...  6.2061 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1991...  Training loss: 1.4216...  6.2126 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1992...  Training loss: 1.4352...  6.1845 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1993...  Training loss: 1.4595...  6.3648 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1994...  Training loss: 1.4414...  6.2662 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1995...  Training loss: 1.4423...  6.2703 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1996...  Training loss: 1.4291...  6.2859 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1997...  Training loss: 1.4339...  6.4164 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1998...  Training loss: 1.4114...  6.2535 sec/batch\n",
      "Epoch: 10/20...  Training Step: 1999...  Training loss: 1.4393...  6.2313 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2000...  Training loss: 1.4184...  6.2098 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2001...  Training loss: 1.4306...  6.2594 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2002...  Training loss: 1.4186...  6.1829 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2003...  Training loss: 1.4247...  6.2494 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2004...  Training loss: 1.4214...  6.2229 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2005...  Training loss: 1.4305...  6.2905 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2006...  Training loss: 1.4585...  6.2494 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2007...  Training loss: 1.4242...  6.2781 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2008...  Training loss: 1.4241...  6.3044 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2009...  Training loss: 1.4176...  6.2606 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2010...  Training loss: 1.4218...  6.2803 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2011...  Training loss: 1.4286...  6.3085 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2012...  Training loss: 1.4375...  6.2737 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2013...  Training loss: 1.4096...  6.3922 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2014...  Training loss: 1.4187...  6.3481 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2015...  Training loss: 1.3956...  6.2255 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2016...  Training loss: 1.4274...  6.2750 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2017...  Training loss: 1.4158...  6.3016 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2018...  Training loss: 1.4048...  6.3290 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2019...  Training loss: 1.4416...  6.2564 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2020...  Training loss: 1.4052...  6.2833 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2021...  Training loss: 1.4050...  6.2756 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2022...  Training loss: 1.4063...  6.3037 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2023...  Training loss: 1.3933...  6.2555 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2024...  Training loss: 1.4060...  6.2429 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2025...  Training loss: 1.3816...  6.2191 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2026...  Training loss: 1.4098...  6.2565 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2027...  Training loss: 1.4300...  6.2248 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2028...  Training loss: 1.4617...  6.2530 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2029...  Training loss: 1.4216...  6.2880 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2030...  Training loss: 1.4410...  6.2695 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2031...  Training loss: 1.4094...  6.2620 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2032...  Training loss: 1.4394...  6.2190 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2033...  Training loss: 1.4424...  6.2071 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2034...  Training loss: 1.4384...  6.2730 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2035...  Training loss: 1.4514...  6.1910 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2036...  Training loss: 1.4115...  6.1877 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2037...  Training loss: 1.4699...  6.2361 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2038...  Training loss: 1.4491...  6.2279 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2039...  Training loss: 1.4227...  6.2767 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2040...  Training loss: 1.4419...  6.2390 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2041...  Training loss: 1.4036...  6.2946 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2042...  Training loss: 1.4593...  6.2143 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2043...  Training loss: 1.4162...  6.2752 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2044...  Training loss: 1.3971...  6.2429 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2045...  Training loss: 1.4410...  6.2258 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2046...  Training loss: 1.4363...  6.1829 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2047...  Training loss: 1.4588...  6.2197 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2048...  Training loss: 1.4278...  6.1914 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2049...  Training loss: 1.4224...  6.2777 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2050...  Training loss: 1.4426...  6.2668 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2051...  Training loss: 1.4124...  6.2348 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2052...  Training loss: 1.4273...  6.2151 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2053...  Training loss: 1.4473...  6.2316 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2054...  Training loss: 1.4321...  6.2008 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2055...  Training loss: 1.4400...  6.2290 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2056...  Training loss: 1.4346...  6.2302 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2057...  Training loss: 1.4397...  6.2120 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2058...  Training loss: 1.4356...  6.2261 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2059...  Training loss: 1.4168...  6.2511 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2060...  Training loss: 1.4692...  6.1964 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2061...  Training loss: 1.4258...  6.2480 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2062...  Training loss: 1.4170...  6.1978 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2063...  Training loss: 1.4166...  6.2289 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2064...  Training loss: 1.4613...  6.2674 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2065...  Training loss: 1.4441...  6.1639 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2066...  Training loss: 1.4125...  6.2728 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2067...  Training loss: 1.4213...  6.2618 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2068...  Training loss: 1.4651...  6.2303 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2069...  Training loss: 1.4399...  6.2766 sec/batch\n",
      "Epoch: 10/20...  Training Step: 2070...  Training loss: 1.4460...  6.1934 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2071...  Training loss: 1.5144...  6.1985 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2072...  Training loss: 1.4462...  6.2107 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2073...  Training loss: 1.4387...  6.1798 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2074...  Training loss: 1.4767...  6.2217 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2075...  Training loss: 1.4431...  6.2099 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2076...  Training loss: 1.4367...  6.2092 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2077...  Training loss: 1.4447...  6.1821 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2078...  Training loss: 1.4462...  6.2085 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2079...  Training loss: 1.4506...  6.3079 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2080...  Training loss: 1.4358...  6.2697 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2081...  Training loss: 1.4367...  6.2100 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2082...  Training loss: 1.4175...  6.2339 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2083...  Training loss: 1.4653...  6.1832 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2084...  Training loss: 1.4409...  6.1912 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2085...  Training loss: 1.4305...  6.2224 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2086...  Training loss: 1.3967...  6.1758 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2087...  Training loss: 1.4071...  6.2061 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2088...  Training loss: 1.4274...  6.2243 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2089...  Training loss: 1.4433...  6.2212 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2090...  Training loss: 1.4319...  6.2444 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2091...  Training loss: 1.4250...  6.2443 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2092...  Training loss: 1.4263...  6.2289 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2093...  Training loss: 1.4099...  6.2078 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2094...  Training loss: 1.4171...  6.2042 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2095...  Training loss: 1.4198...  6.2388 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2096...  Training loss: 1.4425...  6.2404 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2097...  Training loss: 1.4275...  6.1742 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2098...  Training loss: 1.4191...  6.2515 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2099...  Training loss: 1.4035...  6.1973 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2100...  Training loss: 1.4260...  6.2620 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2101...  Training loss: 1.3950...  6.2284 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2102...  Training loss: 1.4269...  6.2216 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2103...  Training loss: 1.4500...  6.1713 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2104...  Training loss: 1.4410...  6.2426 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2105...  Training loss: 1.4376...  6.2186 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2106...  Training loss: 1.4071...  6.1969 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2107...  Training loss: 1.4326...  6.2640 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2108...  Training loss: 1.4088...  6.2719 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2109...  Training loss: 1.4122...  6.2637 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2110...  Training loss: 1.4174...  6.2464 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2111...  Training loss: 1.4135...  6.2535 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2112...  Training loss: 1.3996...  6.1618 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2113...  Training loss: 1.4141...  6.2327 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2114...  Training loss: 1.3887...  6.2560 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2115...  Training loss: 1.4084...  6.2139 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2116...  Training loss: 1.4081...  6.2231 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2117...  Training loss: 1.4188...  6.2087 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2118...  Training loss: 1.4181...  6.2321 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2119...  Training loss: 1.4200...  6.2477 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2120...  Training loss: 1.4383...  6.2707 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2121...  Training loss: 1.4113...  6.2361 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2122...  Training loss: 1.4005...  6.2055 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2123...  Training loss: 1.4122...  6.2002 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2124...  Training loss: 1.4213...  6.2126 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2125...  Training loss: 1.3774...  6.1740 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2126...  Training loss: 1.4202...  6.2796 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2127...  Training loss: 1.3998...  6.1921 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2128...  Training loss: 1.3972...  6.2384 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2129...  Training loss: 1.4276...  6.1638 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2130...  Training loss: 1.4228...  6.2680 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2131...  Training loss: 1.4068...  6.2407 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2132...  Training loss: 1.3926...  6.2354 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2133...  Training loss: 1.4394...  6.2499 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2134...  Training loss: 1.4134...  6.2458 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2135...  Training loss: 1.4364...  6.2644 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2136...  Training loss: 1.4767...  6.2339 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2137...  Training loss: 1.4204...  6.2453 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2138...  Training loss: 1.4370...  6.2077 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2139...  Training loss: 1.4342...  6.1971 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2140...  Training loss: 1.4233...  6.2972 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2141...  Training loss: 1.4462...  6.2556 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2142...  Training loss: 1.3967...  6.2012 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2143...  Training loss: 1.4167...  6.2299 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2144...  Training loss: 1.4007...  6.2268 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2145...  Training loss: 1.3861...  6.1666 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2146...  Training loss: 1.4118...  6.1945 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2147...  Training loss: 1.3942...  6.2286 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2148...  Training loss: 1.3846...  6.1772 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2149...  Training loss: 1.4173...  6.2270 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2150...  Training loss: 1.4115...  6.2306 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2151...  Training loss: 1.4060...  6.3022 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2152...  Training loss: 1.4111...  6.2470 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2153...  Training loss: 1.4136...  6.2625 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2154...  Training loss: 1.4156...  6.3169 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2155...  Training loss: 1.4063...  6.2234 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2156...  Training loss: 1.3933...  6.3161 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2157...  Training loss: 1.4164...  6.4046 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2158...  Training loss: 1.4217...  6.1790 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2159...  Training loss: 1.4213...  6.1848 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2160...  Training loss: 1.4100...  6.2412 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2161...  Training loss: 1.4032...  6.2565 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2162...  Training loss: 1.3838...  6.2470 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2163...  Training loss: 1.4197...  6.3427 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2164...  Training loss: 1.3812...  6.2735 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2165...  Training loss: 1.4088...  6.3027 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2166...  Training loss: 1.3955...  6.2720 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2167...  Training loss: 1.4229...  6.2164 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2168...  Training loss: 1.3684...  6.2525 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2169...  Training loss: 1.3720...  6.2486 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2170...  Training loss: 1.4139...  6.2579 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2171...  Training loss: 1.4164...  6.2905 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2172...  Training loss: 1.4197...  6.1937 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2173...  Training loss: 1.3814...  6.2423 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2174...  Training loss: 1.4172...  6.2559 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2175...  Training loss: 1.4039...  6.2615 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2176...  Training loss: 1.4008...  6.2334 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2177...  Training loss: 1.4201...  6.2463 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2178...  Training loss: 1.4198...  6.2724 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2179...  Training loss: 1.4059...  6.2700 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2180...  Training loss: 1.4364...  6.2921 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2181...  Training loss: 1.4399...  6.2124 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2182...  Training loss: 1.3969...  6.2452 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2183...  Training loss: 1.3959...  6.1860 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2184...  Training loss: 1.3959...  6.2126 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2185...  Training loss: 1.4168...  6.2293 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2186...  Training loss: 1.4301...  6.1771 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2187...  Training loss: 1.4357...  6.1804 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2188...  Training loss: 1.4298...  6.2576 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2189...  Training loss: 1.4172...  6.2659 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2190...  Training loss: 1.4050...  6.2654 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2191...  Training loss: 1.3765...  6.2184 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2192...  Training loss: 1.4126...  6.2201 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2193...  Training loss: 1.3891...  6.2279 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2194...  Training loss: 1.3990...  6.2669 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2195...  Training loss: 1.4144...  6.2529 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2196...  Training loss: 1.3871...  6.1670 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2197...  Training loss: 1.4014...  6.2536 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2198...  Training loss: 1.4053...  6.2011 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2199...  Training loss: 1.4074...  6.1847 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2200...  Training loss: 1.4380...  6.2650 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2201...  Training loss: 1.4163...  6.2325 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2202...  Training loss: 1.4125...  6.2451 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2203...  Training loss: 1.3994...  6.2623 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2204...  Training loss: 1.4052...  6.2527 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2205...  Training loss: 1.3936...  6.2248 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2206...  Training loss: 1.4072...  6.2287 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2207...  Training loss: 1.4010...  6.2126 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2208...  Training loss: 1.4095...  6.2256 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2209...  Training loss: 1.3910...  6.1729 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2210...  Training loss: 1.3997...  6.2828 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2211...  Training loss: 1.3838...  6.2339 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2212...  Training loss: 1.3973...  6.1993 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2213...  Training loss: 1.4327...  6.2118 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2214...  Training loss: 1.3975...  6.1999 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2215...  Training loss: 1.4032...  6.2575 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2216...  Training loss: 1.3860...  6.2896 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2217...  Training loss: 1.4062...  6.1969 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2218...  Training loss: 1.4059...  6.2402 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2219...  Training loss: 1.4084...  6.2280 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2220...  Training loss: 1.3868...  6.2363 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2221...  Training loss: 1.3840...  6.2200 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2222...  Training loss: 1.3701...  6.1913 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2223...  Training loss: 1.3987...  6.2546 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2224...  Training loss: 1.3742...  6.2454 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2225...  Training loss: 1.3807...  6.2216 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2226...  Training loss: 1.3983...  6.2704 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2227...  Training loss: 1.3752...  6.2165 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2228...  Training loss: 1.3830...  6.2261 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2229...  Training loss: 1.3862...  6.2579 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2230...  Training loss: 1.3683...  6.2984 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2231...  Training loss: 1.3720...  6.2258 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2232...  Training loss: 1.3582...  6.2486 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2233...  Training loss: 1.3959...  6.3936 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2234...  Training loss: 1.3961...  6.3719 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2235...  Training loss: 1.4364...  6.3119 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2236...  Training loss: 1.3913...  6.2705 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2237...  Training loss: 1.4181...  6.1982 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2238...  Training loss: 1.3830...  6.2715 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2239...  Training loss: 1.4094...  6.2045 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2240...  Training loss: 1.4217...  6.1808 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2241...  Training loss: 1.4133...  6.2368 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2242...  Training loss: 1.4133...  6.2825 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2243...  Training loss: 1.3889...  6.2162 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2244...  Training loss: 1.4495...  6.2317 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2245...  Training loss: 1.4209...  6.2434 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2246...  Training loss: 1.4070...  6.2444 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2247...  Training loss: 1.4057...  6.2484 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2248...  Training loss: 1.3773...  6.2031 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2249...  Training loss: 1.4343...  6.1793 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2250...  Training loss: 1.3917...  6.2172 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2251...  Training loss: 1.3764...  6.2373 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2252...  Training loss: 1.4165...  6.2044 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2253...  Training loss: 1.4170...  6.2708 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2254...  Training loss: 1.4232...  6.2396 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2255...  Training loss: 1.4173...  6.2609 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2256...  Training loss: 1.3977...  6.1825 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2257...  Training loss: 1.4013...  6.2478 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2258...  Training loss: 1.3883...  6.2080 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2259...  Training loss: 1.3919...  6.2128 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2260...  Training loss: 1.4289...  6.1994 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2261...  Training loss: 1.4083...  6.2456 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2262...  Training loss: 1.4111...  6.2922 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2263...  Training loss: 1.4118...  6.1875 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2264...  Training loss: 1.4091...  6.2330 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2265...  Training loss: 1.4020...  6.2432 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2266...  Training loss: 1.3874...  6.2170 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2267...  Training loss: 1.4414...  6.2185 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2268...  Training loss: 1.4076...  6.2479 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2269...  Training loss: 1.3880...  6.2237 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2270...  Training loss: 1.3876...  6.2391 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2271...  Training loss: 1.4350...  6.2661 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2272...  Training loss: 1.4169...  6.2947 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2273...  Training loss: 1.3842...  6.2620 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2274...  Training loss: 1.4041...  6.2731 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2275...  Training loss: 1.4348...  6.3024 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2276...  Training loss: 1.4041...  6.3017 sec/batch\n",
      "Epoch: 11/20...  Training Step: 2277...  Training loss: 1.4216...  6.2828 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2278...  Training loss: 1.4923...  6.2854 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2279...  Training loss: 1.4124...  6.2717 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2280...  Training loss: 1.4205...  6.2898 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2281...  Training loss: 1.4628...  6.2710 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2282...  Training loss: 1.4143...  6.2528 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2283...  Training loss: 1.4129...  6.2621 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2284...  Training loss: 1.4182...  6.2703 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2285...  Training loss: 1.4101...  6.3466 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2286...  Training loss: 1.4332...  6.3030 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2287...  Training loss: 1.4216...  6.3303 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2288...  Training loss: 1.4125...  6.2883 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2289...  Training loss: 1.3839...  6.2793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2290...  Training loss: 1.4291...  6.2653 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2291...  Training loss: 1.4202...  6.3347 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2292...  Training loss: 1.4054...  6.2454 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2293...  Training loss: 1.3740...  6.2007 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2294...  Training loss: 1.3810...  6.3062 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2295...  Training loss: 1.4065...  6.2891 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2296...  Training loss: 1.4096...  6.2352 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2297...  Training loss: 1.4058...  6.2197 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2298...  Training loss: 1.3929...  6.2529 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2299...  Training loss: 1.3936...  6.2599 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2300...  Training loss: 1.3884...  6.2061 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2301...  Training loss: 1.4048...  6.2680 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2302...  Training loss: 1.3974...  6.2769 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2303...  Training loss: 1.4186...  6.2800 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2304...  Training loss: 1.4044...  6.2576 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2305...  Training loss: 1.3876...  6.2845 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2306...  Training loss: 1.3833...  6.2937 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2307...  Training loss: 1.4012...  6.2940 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2308...  Training loss: 1.3675...  6.2460 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2309...  Training loss: 1.4009...  6.2986 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2310...  Training loss: 1.4171...  6.2464 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2311...  Training loss: 1.4097...  6.2277 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2312...  Training loss: 1.4078...  6.2782 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2313...  Training loss: 1.3845...  6.2545 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2314...  Training loss: 1.4010...  6.2936 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2315...  Training loss: 1.3835...  6.2468 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2316...  Training loss: 1.3811...  6.3146 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2317...  Training loss: 1.3991...  6.2792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2318...  Training loss: 1.3889...  6.2219 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2319...  Training loss: 1.3769...  6.2404 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2320...  Training loss: 1.3772...  6.2362 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2321...  Training loss: 1.3591...  6.2893 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2322...  Training loss: 1.3712...  6.2423 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2323...  Training loss: 1.3833...  6.2028 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2324...  Training loss: 1.3857...  6.2548 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2325...  Training loss: 1.3971...  6.2186 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2326...  Training loss: 1.3994...  6.2480 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2327...  Training loss: 1.4051...  6.2775 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2328...  Training loss: 1.3830...  6.3060 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2329...  Training loss: 1.3783...  6.2714 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2330...  Training loss: 1.3768...  6.3060 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2331...  Training loss: 1.3953...  6.2408 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2332...  Training loss: 1.3552...  6.2867 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2333...  Training loss: 1.3963...  6.1910 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2334...  Training loss: 1.3829...  6.2318 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2335...  Training loss: 1.3751...  6.2515 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2336...  Training loss: 1.4084...  6.2099 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2337...  Training loss: 1.4083...  6.2412 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2338...  Training loss: 1.3779...  6.2905 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2339...  Training loss: 1.3680...  6.2872 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2340...  Training loss: 1.4006...  6.2680 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2341...  Training loss: 1.3846...  6.2659 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2342...  Training loss: 1.4125...  6.2111 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2343...  Training loss: 1.4497...  6.2730 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2344...  Training loss: 1.3965...  6.2639 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2345...  Training loss: 1.4125...  6.2815 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2346...  Training loss: 1.4054...  6.2890 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2347...  Training loss: 1.3915...  6.2788 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2348...  Training loss: 1.4080...  6.2530 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2349...  Training loss: 1.3688...  6.2900 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2350...  Training loss: 1.3852...  6.2553 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2351...  Training loss: 1.3723...  6.2629 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2352...  Training loss: 1.3646...  6.2701 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2353...  Training loss: 1.3900...  6.2844 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2354...  Training loss: 1.3742...  6.2622 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2355...  Training loss: 1.3541...  6.2603 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2356...  Training loss: 1.4006...  6.2528 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2357...  Training loss: 1.3739...  6.2638 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2358...  Training loss: 1.3875...  6.2250 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2359...  Training loss: 1.3945...  6.2793 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2360...  Training loss: 1.3863...  6.3039 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2361...  Training loss: 1.4008...  6.3284 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2362...  Training loss: 1.3804...  6.2806 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2363...  Training loss: 1.3845...  6.2551 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2364...  Training loss: 1.3901...  6.2822 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2365...  Training loss: 1.3917...  6.2726 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2366...  Training loss: 1.3877...  6.2641 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2367...  Training loss: 1.3946...  6.2594 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2368...  Training loss: 1.3749...  6.2990 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2369...  Training loss: 1.3643...  6.2182 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2370...  Training loss: 1.4105...  6.2781 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2371...  Training loss: 1.3595...  6.2595 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2372...  Training loss: 1.3850...  6.3140 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2373...  Training loss: 1.3714...  6.2847 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2374...  Training loss: 1.4005...  6.2591 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2375...  Training loss: 1.3556...  6.3430 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2376...  Training loss: 1.3438...  6.3079 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2377...  Training loss: 1.3907...  6.2612 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2378...  Training loss: 1.4007...  6.2704 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2379...  Training loss: 1.3923...  6.2531 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2380...  Training loss: 1.3573...  6.2714 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2381...  Training loss: 1.3932...  6.2492 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2382...  Training loss: 1.3745...  6.2229 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2383...  Training loss: 1.3654...  6.2252 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2384...  Training loss: 1.3808...  6.2061 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2385...  Training loss: 1.3991...  6.3074 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2386...  Training loss: 1.3930...  6.3432 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2387...  Training loss: 1.4290...  6.1717 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2388...  Training loss: 1.4212...  6.2185 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2389...  Training loss: 1.3726...  6.2556 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2390...  Training loss: 1.3757...  6.2171 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2391...  Training loss: 1.3671...  6.2309 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2392...  Training loss: 1.4017...  6.3092 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2393...  Training loss: 1.4063...  6.2636 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2394...  Training loss: 1.4050...  6.3139 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2395...  Training loss: 1.3966...  6.3262 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2396...  Training loss: 1.3991...  6.2939 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2397...  Training loss: 1.3835...  6.2102 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2398...  Training loss: 1.3559...  6.3215 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2399...  Training loss: 1.3914...  6.2792 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2400...  Training loss: 1.3615...  6.2402 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2401...  Training loss: 1.3748...  6.2356 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2402...  Training loss: 1.3927...  6.2060 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2403...  Training loss: 1.3577...  6.2020 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2404...  Training loss: 1.3811...  6.2150 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2405...  Training loss: 1.3782...  6.2158 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2406...  Training loss: 1.3806...  6.2025 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2407...  Training loss: 1.4067...  6.2847 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2408...  Training loss: 1.3992...  6.2427 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2409...  Training loss: 1.3867...  6.2814 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2410...  Training loss: 1.3804...  6.3333 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2411...  Training loss: 1.3819...  6.2220 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2412...  Training loss: 1.3711...  6.2902 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2413...  Training loss: 1.3748...  6.2406 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2414...  Training loss: 1.3745...  6.2632 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2415...  Training loss: 1.3826...  6.2612 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2416...  Training loss: 1.3678...  6.2184 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2417...  Training loss: 1.3784...  6.2626 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2418...  Training loss: 1.3576...  6.3257 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2419...  Training loss: 1.3847...  6.3004 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2420...  Training loss: 1.4061...  6.2785 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2421...  Training loss: 1.3700...  6.2136 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2422...  Training loss: 1.3771...  6.2294 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2423...  Training loss: 1.3576...  6.2061 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2424...  Training loss: 1.3771...  6.2035 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2425...  Training loss: 1.3808...  6.2456 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2426...  Training loss: 1.3819...  6.3034 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2427...  Training loss: 1.3606...  6.2289 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2428...  Training loss: 1.3620...  6.2584 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2429...  Training loss: 1.3415...  6.2506 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2430...  Training loss: 1.3660...  6.2976 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2431...  Training loss: 1.3666...  6.2726 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2432...  Training loss: 1.3552...  6.2427 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2433...  Training loss: 1.3833...  6.2403 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2434...  Training loss: 1.3453...  6.2567 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2435...  Training loss: 1.3575...  6.2896 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2436...  Training loss: 1.3553...  6.2662 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2437...  Training loss: 1.3343...  6.2192 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2438...  Training loss: 1.3531...  6.2417 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2439...  Training loss: 1.3334...  6.2425 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2440...  Training loss: 1.3650...  6.2052 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2441...  Training loss: 1.3732...  6.2173 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2442...  Training loss: 1.4095...  6.1870 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2443...  Training loss: 1.3743...  6.2162 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2444...  Training loss: 1.4024...  6.2362 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2445...  Training loss: 1.3632...  6.2073 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2446...  Training loss: 1.3974...  6.2523 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2447...  Training loss: 1.4020...  6.2406 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2448...  Training loss: 1.3909...  6.2833 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2449...  Training loss: 1.4012...  6.2896 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2450...  Training loss: 1.3620...  6.3047 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2451...  Training loss: 1.4301...  6.2849 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2452...  Training loss: 1.4040...  6.2104 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2453...  Training loss: 1.3800...  6.2879 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2454...  Training loss: 1.3905...  6.1882 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2455...  Training loss: 1.3555...  6.1766 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2456...  Training loss: 1.4127...  6.2367 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2457...  Training loss: 1.3710...  6.2743 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2458...  Training loss: 1.3387...  6.2059 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2459...  Training loss: 1.3960...  6.2239 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2460...  Training loss: 1.3856...  6.2077 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2461...  Training loss: 1.3996...  6.1896 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2462...  Training loss: 1.3845...  6.2607 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2463...  Training loss: 1.3826...  6.2706 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2464...  Training loss: 1.3914...  6.2103 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2465...  Training loss: 1.3633...  6.2993 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2466...  Training loss: 1.3665...  6.3098 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2467...  Training loss: 1.3905...  6.2322 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2468...  Training loss: 1.3820...  6.1710 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2469...  Training loss: 1.3879...  6.2260 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2470...  Training loss: 1.3799...  6.2509 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2471...  Training loss: 1.3874...  6.1764 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2472...  Training loss: 1.3789...  6.2620 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2473...  Training loss: 1.3636...  6.2239 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2474...  Training loss: 1.4273...  6.2126 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2475...  Training loss: 1.3736...  6.2464 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2476...  Training loss: 1.3636...  6.2372 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2477...  Training loss: 1.3694...  6.2682 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2478...  Training loss: 1.4152...  6.2521 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2479...  Training loss: 1.3979...  6.2328 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2480...  Training loss: 1.3591...  6.2961 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2481...  Training loss: 1.3797...  6.2732 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2482...  Training loss: 1.4129...  6.2544 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2483...  Training loss: 1.3934...  6.2222 sec/batch\n",
      "Epoch: 12/20...  Training Step: 2484...  Training loss: 1.3986...  6.2412 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2485...  Training loss: 1.4673...  6.2428 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2486...  Training loss: 1.3753...  6.2272 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2487...  Training loss: 1.3891...  6.1908 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2488...  Training loss: 1.4202...  6.2136 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2489...  Training loss: 1.3875...  6.2115 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2490...  Training loss: 1.3877...  6.2377 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2491...  Training loss: 1.3922...  6.2625 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2492...  Training loss: 1.3988...  6.2323 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2493...  Training loss: 1.4122...  6.2515 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2494...  Training loss: 1.3893...  6.2206 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2495...  Training loss: 1.3971...  6.2556 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2496...  Training loss: 1.3562...  6.3210 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2497...  Training loss: 1.3987...  6.2545 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2498...  Training loss: 1.3815...  6.2508 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2499...  Training loss: 1.3890...  6.2530 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2500...  Training loss: 1.3482...  6.2391 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2501...  Training loss: 1.3582...  6.2567 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2502...  Training loss: 1.3745...  6.1709 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2503...  Training loss: 1.3866...  6.2322 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2504...  Training loss: 1.3758...  6.2636 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2505...  Training loss: 1.3675...  6.2281 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2506...  Training loss: 1.3750...  6.2836 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2507...  Training loss: 1.3671...  6.2661 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2508...  Training loss: 1.3720...  6.3337 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2509...  Training loss: 1.3565...  6.2565 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2510...  Training loss: 1.3929...  6.3112 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2511...  Training loss: 1.3765...  6.2022 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2512...  Training loss: 1.3671...  6.1838 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2513...  Training loss: 1.3636...  6.2272 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2514...  Training loss: 1.3818...  6.2253 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2515...  Training loss: 1.3465...  6.2178 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2516...  Training loss: 1.3724...  6.1808 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2517...  Training loss: 1.3938...  6.2597 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2518...  Training loss: 1.3845...  6.2427 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2519...  Training loss: 1.3855...  6.2369 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2520...  Training loss: 1.3716...  6.2717 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2521...  Training loss: 1.3803...  6.2992 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2522...  Training loss: 1.3667...  6.2679 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2523...  Training loss: 1.3524...  6.3421 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2524...  Training loss: 1.3790...  6.3240 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2525...  Training loss: 1.3724...  6.2869 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2526...  Training loss: 1.3618...  6.3123 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2527...  Training loss: 1.3650...  6.3692 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2528...  Training loss: 1.3399...  6.2729 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2529...  Training loss: 1.3519...  6.2784 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2530...  Training loss: 1.3639...  6.2909 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2531...  Training loss: 1.3612...  6.3050 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2532...  Training loss: 1.3687...  6.2507 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2533...  Training loss: 1.3712...  6.2901 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2534...  Training loss: 1.3936...  6.2897 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2535...  Training loss: 1.3579...  6.3284 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2536...  Training loss: 1.3584...  6.2850 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2537...  Training loss: 1.3663...  6.3076 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2538...  Training loss: 1.3663...  6.3425 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2539...  Training loss: 1.3345...  6.3337 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2540...  Training loss: 1.3710...  6.2926 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2541...  Training loss: 1.3593...  6.3150 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2542...  Training loss: 1.3564...  6.2836 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2543...  Training loss: 1.3751...  6.2284 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2544...  Training loss: 1.3868...  6.1978 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2545...  Training loss: 1.3623...  6.2821 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2546...  Training loss: 1.3511...  6.3468 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2547...  Training loss: 1.3860...  6.3214 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2548...  Training loss: 1.3670...  6.3177 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2549...  Training loss: 1.3899...  6.2950 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2550...  Training loss: 1.4258...  6.2705 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2551...  Training loss: 1.3788...  6.3202 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2552...  Training loss: 1.3912...  6.3551 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2553...  Training loss: 1.3852...  6.2826 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2554...  Training loss: 1.3679...  6.2882 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2555...  Training loss: 1.3903...  6.2147 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2556...  Training loss: 1.3451...  6.2085 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2557...  Training loss: 1.3611...  6.2142 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2558...  Training loss: 1.3460...  6.2202 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2559...  Training loss: 1.3397...  6.1974 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2560...  Training loss: 1.3736...  6.2334 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2561...  Training loss: 1.3450...  6.2479 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2562...  Training loss: 1.3408...  6.2456 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2563...  Training loss: 1.3753...  6.2567 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2564...  Training loss: 1.3631...  6.2041 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2565...  Training loss: 1.3606...  6.2500 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2566...  Training loss: 1.3638...  6.2577 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2567...  Training loss: 1.3672...  6.2590 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2568...  Training loss: 1.3758...  6.2958 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2569...  Training loss: 1.3640...  6.2528 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2570...  Training loss: 1.3627...  6.2451 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2571...  Training loss: 1.3733...  6.2212 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2572...  Training loss: 1.3706...  6.2754 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2573...  Training loss: 1.3661...  6.2073 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2574...  Training loss: 1.3729...  6.2479 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2575...  Training loss: 1.3477...  6.2136 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2576...  Training loss: 1.3418...  6.2453 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2577...  Training loss: 1.3793...  6.1901 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2578...  Training loss: 1.3430...  6.2281 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2579...  Training loss: 1.3614...  6.2386 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2580...  Training loss: 1.3517...  6.2766 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2581...  Training loss: 1.3771...  6.2389 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2582...  Training loss: 1.3301...  6.2925 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2583...  Training loss: 1.3221...  6.3382 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2584...  Training loss: 1.3690...  6.2561 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2585...  Training loss: 1.3710...  6.2445 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2586...  Training loss: 1.3712...  6.2080 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2587...  Training loss: 1.3324...  6.2384 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2588...  Training loss: 1.3718...  6.2181 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2589...  Training loss: 1.3586...  6.2701 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2590...  Training loss: 1.3526...  6.2387 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2591...  Training loss: 1.3724...  6.2312 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2592...  Training loss: 1.3785...  6.2274 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2593...  Training loss: 1.3672...  6.2654 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2594...  Training loss: 1.3992...  6.2721 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2595...  Training loss: 1.3954...  6.2660 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2596...  Training loss: 1.3450...  6.2620 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2597...  Training loss: 1.3366...  6.3077 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2598...  Training loss: 1.3482...  6.2750 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2599...  Training loss: 1.3875...  6.2363 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2600...  Training loss: 1.3814...  6.2346 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2601...  Training loss: 1.3820...  6.2662 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2602...  Training loss: 1.3834...  6.2511 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2603...  Training loss: 1.3794...  6.2030 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2604...  Training loss: 1.3658...  6.1773 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2605...  Training loss: 1.3376...  6.2986 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2606...  Training loss: 1.3747...  6.2337 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2607...  Training loss: 1.3396...  6.2412 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2608...  Training loss: 1.3568...  6.2582 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2609...  Training loss: 1.3810...  6.2052 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2610...  Training loss: 1.3334...  6.2685 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2611...  Training loss: 1.3661...  6.2390 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2612...  Training loss: 1.3537...  6.1730 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2613...  Training loss: 1.3682...  6.2099 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2614...  Training loss: 1.3923...  6.2293 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2615...  Training loss: 1.3672...  6.2654 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2616...  Training loss: 1.3627...  6.2258 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2617...  Training loss: 1.3576...  6.2225 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2618...  Training loss: 1.3620...  6.2139 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2619...  Training loss: 1.3533...  6.2328 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2620...  Training loss: 1.3603...  6.2337 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2621...  Training loss: 1.3491...  6.2278 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2622...  Training loss: 1.3614...  6.2780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2623...  Training loss: 1.3614...  6.3853 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2624...  Training loss: 1.3573...  6.5156 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2625...  Training loss: 1.3421...  6.4424 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2626...  Training loss: 1.3577...  6.4667 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2627...  Training loss: 1.3838...  6.4777 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2628...  Training loss: 1.3498...  6.3556 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2629...  Training loss: 1.3519...  6.4523 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2630...  Training loss: 1.3489...  6.3328 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2631...  Training loss: 1.3498...  6.1669 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2632...  Training loss: 1.3544...  6.2465 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2633...  Training loss: 1.3665...  6.2058 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2634...  Training loss: 1.3526...  6.2459 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2635...  Training loss: 1.3388...  6.2395 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2636...  Training loss: 1.3259...  6.2756 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2637...  Training loss: 1.3414...  6.2616 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2638...  Training loss: 1.3437...  6.2677 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2639...  Training loss: 1.3363...  6.2900 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2640...  Training loss: 1.3628...  6.2701 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2641...  Training loss: 1.3363...  6.2128 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2642...  Training loss: 1.3329...  6.2015 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2643...  Training loss: 1.3365...  6.2830 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2644...  Training loss: 1.3217...  6.1822 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2645...  Training loss: 1.3309...  6.2174 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2646...  Training loss: 1.3164...  6.2213 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2647...  Training loss: 1.3471...  6.2466 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2648...  Training loss: 1.3509...  6.2593 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2649...  Training loss: 1.4020...  6.2489 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2650...  Training loss: 1.3562...  6.2315 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2651...  Training loss: 1.3894...  6.2406 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2652...  Training loss: 1.3376...  6.3405 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2653...  Training loss: 1.3718...  6.3145 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2654...  Training loss: 1.3812...  6.3064 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2655...  Training loss: 1.3742...  6.1982 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2656...  Training loss: 1.3810...  6.1927 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2657...  Training loss: 1.3496...  6.2417 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2658...  Training loss: 1.4060...  6.2047 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2659...  Training loss: 1.3832...  6.2398 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2660...  Training loss: 1.3571...  6.2850 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2661...  Training loss: 1.3713...  6.2286 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2662...  Training loss: 1.3359...  6.2417 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2663...  Training loss: 1.3912...  6.2093 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2664...  Training loss: 1.3555...  6.2363 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2665...  Training loss: 1.3251...  6.2498 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2666...  Training loss: 1.3720...  6.2617 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2667...  Training loss: 1.3740...  6.2963 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2668...  Training loss: 1.3866...  6.2406 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2669...  Training loss: 1.3681...  6.2603 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2670...  Training loss: 1.3586...  6.2772 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2671...  Training loss: 1.3655...  6.2880 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2672...  Training loss: 1.3528...  6.2806 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2673...  Training loss: 1.3494...  6.2158 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2674...  Training loss: 1.3934...  6.2429 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2675...  Training loss: 1.3576...  6.2887 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2676...  Training loss: 1.3712...  6.3124 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2677...  Training loss: 1.3636...  6.2753 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2678...  Training loss: 1.3728...  6.2310 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2679...  Training loss: 1.3667...  6.3096 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2680...  Training loss: 1.3538...  6.2893 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2681...  Training loss: 1.3923...  6.2884 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2682...  Training loss: 1.3546...  6.2340 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2683...  Training loss: 1.3450...  6.2422 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2684...  Training loss: 1.3496...  6.1968 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2685...  Training loss: 1.3938...  6.2780 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2686...  Training loss: 1.3765...  6.1915 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2687...  Training loss: 1.3359...  6.2949 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2688...  Training loss: 1.3602...  6.2805 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2689...  Training loss: 1.3919...  6.2660 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2690...  Training loss: 1.3753...  6.2547 sec/batch\n",
      "Epoch: 13/20...  Training Step: 2691...  Training loss: 1.3750...  6.1881 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2692...  Training loss: 1.4473...  6.1936 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2693...  Training loss: 1.3704...  6.2352 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2694...  Training loss: 1.3713...  6.3055 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2695...  Training loss: 1.4122...  6.2364 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2696...  Training loss: 1.3707...  6.2462 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2697...  Training loss: 1.3652...  6.2694 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2698...  Training loss: 1.3827...  6.2354 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2699...  Training loss: 1.3796...  6.2168 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2700...  Training loss: 1.3821...  6.2256 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2701...  Training loss: 1.3612...  6.1841 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2702...  Training loss: 1.3722...  6.2645 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2703...  Training loss: 1.3423...  6.2028 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2704...  Training loss: 1.3837...  6.2135 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2705...  Training loss: 1.3693...  6.2093 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2706...  Training loss: 1.3650...  6.2244 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2707...  Training loss: 1.3291...  6.2374 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2708...  Training loss: 1.3376...  6.2723 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2709...  Training loss: 1.3572...  6.3155 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2710...  Training loss: 1.3587...  6.2822 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2711...  Training loss: 1.3571...  6.2540 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2712...  Training loss: 1.3464...  6.2091 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2713...  Training loss: 1.3570...  6.2358 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2714...  Training loss: 1.3503...  6.2586 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2715...  Training loss: 1.3517...  6.2111 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2716...  Training loss: 1.3589...  6.2675 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2717...  Training loss: 1.3787...  6.1962 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2718...  Training loss: 1.3709...  6.2125 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2719...  Training loss: 1.3527...  6.2226 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2720...  Training loss: 1.3443...  6.2858 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2721...  Training loss: 1.3594...  6.2286 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2722...  Training loss: 1.3263...  6.2886 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2723...  Training loss: 1.3587...  6.2777 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2724...  Training loss: 1.3825...  6.2216 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2725...  Training loss: 1.3634...  6.2984 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2726...  Training loss: 1.3704...  6.2478 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2727...  Training loss: 1.3486...  6.1835 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2728...  Training loss: 1.3659...  6.2293 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2729...  Training loss: 1.3516...  6.1958 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2730...  Training loss: 1.3444...  6.1940 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2731...  Training loss: 1.3542...  6.2268 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2732...  Training loss: 1.3493...  6.2843 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2733...  Training loss: 1.3394...  6.2095 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2734...  Training loss: 1.3579...  6.2119 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2735...  Training loss: 1.3275...  6.2380 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2736...  Training loss: 1.3360...  6.2690 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2737...  Training loss: 1.3650...  6.2694 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2738...  Training loss: 1.3538...  6.3164 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2739...  Training loss: 1.3458...  6.2749 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2740...  Training loss: 1.3696...  6.2682 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2741...  Training loss: 1.3753...  6.2538 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2742...  Training loss: 1.3522...  6.2365 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2743...  Training loss: 1.3481...  6.2285 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2744...  Training loss: 1.3520...  6.1909 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2745...  Training loss: 1.3475...  6.2210 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2746...  Training loss: 1.3239...  6.2469 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2747...  Training loss: 1.3671...  6.1948 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2748...  Training loss: 1.3479...  6.2155 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2749...  Training loss: 1.3324...  6.3143 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2750...  Training loss: 1.3664...  6.2742 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2751...  Training loss: 1.3675...  6.2742 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2752...  Training loss: 1.3512...  6.2622 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2753...  Training loss: 1.3308...  6.2814 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2754...  Training loss: 1.3608...  6.2603 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2755...  Training loss: 1.3506...  6.1986 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2756...  Training loss: 1.3745...  6.2099 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2757...  Training loss: 1.4096...  6.1880 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2758...  Training loss: 1.3549...  6.2104 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2759...  Training loss: 1.3710...  6.2083 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2760...  Training loss: 1.3715...  6.2131 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2761...  Training loss: 1.3637...  6.1869 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2762...  Training loss: 1.3696...  6.2377 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2763...  Training loss: 1.3233...  6.1970 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2764...  Training loss: 1.3392...  6.2513 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2765...  Training loss: 1.3294...  6.2430 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2766...  Training loss: 1.3292...  6.2947 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2767...  Training loss: 1.3490...  6.3343 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2768...  Training loss: 1.3336...  6.2320 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2769...  Training loss: 1.3206...  6.2303 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2770...  Training loss: 1.3634...  6.2290 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2771...  Training loss: 1.3367...  6.2121 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2772...  Training loss: 1.3442...  6.2127 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2773...  Training loss: 1.3462...  6.2396 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2774...  Training loss: 1.3531...  6.2770 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2775...  Training loss: 1.3441...  6.2553 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2776...  Training loss: 1.3380...  6.2506 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2777...  Training loss: 1.3573...  6.2387 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2778...  Training loss: 1.3483...  6.2550 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2779...  Training loss: 1.3638...  6.2794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2780...  Training loss: 1.3534...  6.2862 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2781...  Training loss: 1.3575...  6.2713 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2782...  Training loss: 1.3277...  6.2225 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2783...  Training loss: 1.3222...  6.2543 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2784...  Training loss: 1.3560...  6.2561 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2785...  Training loss: 1.3290...  6.2060 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2786...  Training loss: 1.3432...  6.2375 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2787...  Training loss: 1.3264...  6.2094 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2788...  Training loss: 1.3524...  6.2375 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2789...  Training loss: 1.3138...  6.2318 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2790...  Training loss: 1.3115...  6.1780 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2791...  Training loss: 1.3514...  6.2119 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2792...  Training loss: 1.3522...  6.3044 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2793...  Training loss: 1.3533...  6.3021 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2794...  Training loss: 1.3137...  6.2305 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2795...  Training loss: 1.3511...  6.2880 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2796...  Training loss: 1.3300...  6.2852 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2797...  Training loss: 1.3229...  6.2556 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2798...  Training loss: 1.3455...  6.2701 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2799...  Training loss: 1.3561...  6.2359 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2800...  Training loss: 1.3534...  6.2689 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2801...  Training loss: 1.3809...  6.2525 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2802...  Training loss: 1.3833...  6.2600 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2803...  Training loss: 1.3335...  6.2347 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2804...  Training loss: 1.3229...  6.2137 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2805...  Training loss: 1.3389...  6.2583 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2806...  Training loss: 1.3635...  6.3153 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2807...  Training loss: 1.3567...  6.3173 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2808...  Training loss: 1.3703...  6.2985 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2809...  Training loss: 1.3652...  6.3264 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2810...  Training loss: 1.3602...  6.2153 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2811...  Training loss: 1.3430...  6.3029 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2812...  Training loss: 1.3203...  6.2181 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2813...  Training loss: 1.3476...  6.2270 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2814...  Training loss: 1.3227...  6.2054 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2815...  Training loss: 1.3274...  6.2047 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2816...  Training loss: 1.3580...  6.1763 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2817...  Training loss: 1.3227...  6.2921 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2818...  Training loss: 1.3358...  6.1966 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2819...  Training loss: 1.3370...  6.2689 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2820...  Training loss: 1.3498...  6.2223 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2821...  Training loss: 1.3714...  6.2585 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2822...  Training loss: 1.3572...  6.2773 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2823...  Training loss: 1.3452...  6.2923 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2824...  Training loss: 1.3433...  6.2694 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2825...  Training loss: 1.3417...  6.2609 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2826...  Training loss: 1.3273...  6.2289 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2827...  Training loss: 1.3423...  6.2264 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2828...  Training loss: 1.3253...  6.2012 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2829...  Training loss: 1.3396...  6.2539 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2830...  Training loss: 1.3344...  6.2161 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2831...  Training loss: 1.3355...  6.2178 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2832...  Training loss: 1.3193...  6.2081 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2833...  Training loss: 1.3369...  6.2571 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2834...  Training loss: 1.3606...  6.2343 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2835...  Training loss: 1.3382...  6.2598 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2836...  Training loss: 1.3300...  6.2119 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2837...  Training loss: 1.3214...  6.2423 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2838...  Training loss: 1.3348...  6.2549 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2839...  Training loss: 1.3444...  6.2947 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2840...  Training loss: 1.3384...  6.2873 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2841...  Training loss: 1.3213...  6.2065 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2842...  Training loss: 1.3218...  6.2678 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2843...  Training loss: 1.3111...  6.2084 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2844...  Training loss: 1.3300...  6.3041 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2845...  Training loss: 1.3300...  6.2201 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2846...  Training loss: 1.3152...  6.2642 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2847...  Training loss: 1.3486...  6.2251 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2848...  Training loss: 1.3156...  6.1876 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2849...  Training loss: 1.3166...  6.2853 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2850...  Training loss: 1.3241...  6.2600 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2851...  Training loss: 1.2965...  6.2458 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2852...  Training loss: 1.3179...  6.3151 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2853...  Training loss: 1.3003...  6.2794 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2854...  Training loss: 1.3343...  6.2784 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2855...  Training loss: 1.3290...  6.2330 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2856...  Training loss: 1.3692...  6.2713 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2857...  Training loss: 1.3294...  6.2056 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2858...  Training loss: 1.3629...  6.1982 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2859...  Training loss: 1.3184...  6.2323 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2860...  Training loss: 1.3501...  6.2582 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2861...  Training loss: 1.3645...  6.2669 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2862...  Training loss: 1.3585...  6.1757 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2863...  Training loss: 1.3700...  6.2558 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2864...  Training loss: 1.3264...  6.2667 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2865...  Training loss: 1.3899...  6.2264 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2866...  Training loss: 1.3694...  6.2150 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2867...  Training loss: 1.3433...  6.2927 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2868...  Training loss: 1.3457...  6.2300 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2869...  Training loss: 1.3148...  6.2434 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2870...  Training loss: 1.3746...  6.1718 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2871...  Training loss: 1.3275...  6.2642 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2872...  Training loss: 1.3137...  6.2653 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2873...  Training loss: 1.3538...  6.2142 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2874...  Training loss: 1.3551...  6.2023 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2875...  Training loss: 1.3693...  6.2342 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2876...  Training loss: 1.3511...  6.1969 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2877...  Training loss: 1.3416...  6.2547 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2878...  Training loss: 1.3484...  6.2741 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2879...  Training loss: 1.3283...  6.2384 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2880...  Training loss: 1.3372...  6.2669 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2881...  Training loss: 1.3669...  6.2504 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2882...  Training loss: 1.3588...  6.1994 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2883...  Training loss: 1.3503...  6.2039 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2884...  Training loss: 1.3466...  6.2604 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2885...  Training loss: 1.3464...  6.2365 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2886...  Training loss: 1.3532...  6.2251 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2887...  Training loss: 1.3279...  6.2112 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2888...  Training loss: 1.3762...  6.2164 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2889...  Training loss: 1.3413...  6.2273 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2890...  Training loss: 1.3309...  6.2026 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2891...  Training loss: 1.3219...  6.2328 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2892...  Training loss: 1.3842...  6.3003 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2893...  Training loss: 1.3621...  6.2531 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2894...  Training loss: 1.3161...  6.3193 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2895...  Training loss: 1.3463...  6.2723 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2896...  Training loss: 1.3804...  6.1932 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2897...  Training loss: 1.3497...  6.2260 sec/batch\n",
      "Epoch: 14/20...  Training Step: 2898...  Training loss: 1.3636...  6.2183 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2899...  Training loss: 1.4448...  6.2393 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2900...  Training loss: 1.3474...  6.2545 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2901...  Training loss: 1.3587...  6.2151 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2902...  Training loss: 1.3891...  6.2275 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2903...  Training loss: 1.3487...  6.2366 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2904...  Training loss: 1.3546...  6.2349 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2905...  Training loss: 1.3618...  6.2371 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2906...  Training loss: 1.3535...  6.3361 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2907...  Training loss: 1.3750...  6.2983 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2908...  Training loss: 1.3560...  6.2766 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2909...  Training loss: 1.3547...  6.2739 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2910...  Training loss: 1.3212...  6.2251 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2911...  Training loss: 1.3747...  6.2160 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2912...  Training loss: 1.3590...  6.2011 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2913...  Training loss: 1.3395...  6.2362 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2914...  Training loss: 1.3210...  6.2290 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2915...  Training loss: 1.3288...  6.2052 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2916...  Training loss: 1.3523...  6.2647 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2917...  Training loss: 1.3466...  6.2067 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2918...  Training loss: 1.3489...  6.1996 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2919...  Training loss: 1.3351...  6.2946 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2920...  Training loss: 1.3375...  6.3223 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2921...  Training loss: 1.3327...  6.2680 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2922...  Training loss: 1.3365...  6.2498 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2923...  Training loss: 1.3369...  6.2294 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2924...  Training loss: 1.3672...  6.1845 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2925...  Training loss: 1.3380...  6.2223 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2926...  Training loss: 1.3319...  6.2459 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2927...  Training loss: 1.3231...  6.2622 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2928...  Training loss: 1.3426...  6.2570 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2929...  Training loss: 1.3081...  6.2948 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2930...  Training loss: 1.3314...  6.2767 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2931...  Training loss: 1.3672...  6.2291 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2932...  Training loss: 1.3514...  6.2314 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2933...  Training loss: 1.3564...  6.2595 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2934...  Training loss: 1.3243...  6.3025 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2935...  Training loss: 1.3442...  6.2163 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2936...  Training loss: 1.3378...  6.2440 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2937...  Training loss: 1.3280...  6.3031 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2938...  Training loss: 1.3385...  6.2309 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2939...  Training loss: 1.3379...  6.1951 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2940...  Training loss: 1.3217...  6.1914 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2941...  Training loss: 1.3289...  6.2764 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2942...  Training loss: 1.3144...  6.2152 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2943...  Training loss: 1.3148...  6.2250 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2944...  Training loss: 1.3261...  6.2329 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2945...  Training loss: 1.3408...  6.2475 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2946...  Training loss: 1.3269...  6.2066 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2947...  Training loss: 1.3386...  6.2426 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2948...  Training loss: 1.3581...  6.2984 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2949...  Training loss: 1.3274...  6.3000 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2950...  Training loss: 1.3170...  6.3035 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2951...  Training loss: 1.3359...  6.2383 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2952...  Training loss: 1.3268...  6.2066 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2953...  Training loss: 1.3019...  6.1843 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2954...  Training loss: 1.3435...  6.2487 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2955...  Training loss: 1.3290...  6.2535 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2956...  Training loss: 1.3165...  6.1963 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2957...  Training loss: 1.3567...  6.2995 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2958...  Training loss: 1.3529...  6.2269 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2959...  Training loss: 1.3261...  6.2333 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2960...  Training loss: 1.3176...  6.2245 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2961...  Training loss: 1.3564...  6.2645 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2962...  Training loss: 1.3254...  6.2698 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2963...  Training loss: 1.3555...  6.2688 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2964...  Training loss: 1.3845...  6.2695 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2965...  Training loss: 1.3410...  6.2546 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2966...  Training loss: 1.3565...  6.2253 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2967...  Training loss: 1.3542...  6.2109 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2968...  Training loss: 1.3304...  6.2491 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2969...  Training loss: 1.3521...  6.2220 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2970...  Training loss: 1.3084...  6.2187 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2971...  Training loss: 1.3245...  6.2168 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2972...  Training loss: 1.3228...  6.2225 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2973...  Training loss: 1.3196...  6.2256 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2974...  Training loss: 1.3429...  6.2718 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2975...  Training loss: 1.3185...  6.2487 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2976...  Training loss: 1.2943...  6.2444 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2977...  Training loss: 1.3418...  6.2550 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2978...  Training loss: 1.3206...  6.2904 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2979...  Training loss: 1.3338...  6.2542 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2980...  Training loss: 1.3319...  6.1725 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2981...  Training loss: 1.3351...  6.2726 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2982...  Training loss: 1.3305...  6.2100 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2983...  Training loss: 1.3186...  6.2666 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2984...  Training loss: 1.3251...  6.2630 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2985...  Training loss: 1.3304...  6.2610 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2986...  Training loss: 1.3325...  6.2085 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2987...  Training loss: 1.3375...  6.2651 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2988...  Training loss: 1.3398...  6.1983 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2989...  Training loss: 1.3073...  6.2559 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2990...  Training loss: 1.3018...  6.2518 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2991...  Training loss: 1.3369...  6.2643 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2992...  Training loss: 1.3086...  6.2280 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2993...  Training loss: 1.3343...  6.2477 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2994...  Training loss: 1.3176...  6.2728 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2995...  Training loss: 1.3350...  6.2496 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2996...  Training loss: 1.2966...  6.2148 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2997...  Training loss: 1.2930...  6.1907 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2998...  Training loss: 1.3429...  6.2319 sec/batch\n",
      "Epoch: 15/20...  Training Step: 2999...  Training loss: 1.3391...  6.2605 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3000...  Training loss: 1.3326...  6.2213 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3001...  Training loss: 1.2997...  6.2100 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3002...  Training loss: 1.3424...  6.2896 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3003...  Training loss: 1.3133...  6.2326 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3004...  Training loss: 1.3169...  6.2395 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3005...  Training loss: 1.3220...  6.3249 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3006...  Training loss: 1.3461...  6.2897 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3007...  Training loss: 1.3361...  6.3308 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3008...  Training loss: 1.3532...  6.2564 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3009...  Training loss: 1.3595...  6.2839 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3010...  Training loss: 1.3086...  6.2456 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3011...  Training loss: 1.3057...  6.2164 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3012...  Training loss: 1.3101...  6.2265 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3013...  Training loss: 1.3443...  6.2046 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3014...  Training loss: 1.3446...  6.1965 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3015...  Training loss: 1.3488...  6.2251 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3016...  Training loss: 1.3476...  6.2283 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3017...  Training loss: 1.3389...  6.2189 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3018...  Training loss: 1.3388...  6.2773 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3019...  Training loss: 1.2962...  6.2898 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3020...  Training loss: 1.3399...  6.3212 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3021...  Training loss: 1.3025...  6.2564 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3022...  Training loss: 1.3188...  6.2688 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3023...  Training loss: 1.3459...  6.2467 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3024...  Training loss: 1.2991...  6.2227 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3025...  Training loss: 1.3318...  6.2167 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3026...  Training loss: 1.3164...  6.2279 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3027...  Training loss: 1.3253...  6.2352 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3028...  Training loss: 1.3500...  6.2054 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3029...  Training loss: 1.3362...  6.2593 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3030...  Training loss: 1.3348...  6.2444 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3031...  Training loss: 1.3280...  6.2712 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3032...  Training loss: 1.3202...  6.2483 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3033...  Training loss: 1.3117...  6.2773 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3034...  Training loss: 1.3218...  6.2930 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3035...  Training loss: 1.3135...  6.2635 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3036...  Training loss: 1.3304...  6.2467 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3037...  Training loss: 1.3229...  6.2188 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3038...  Training loss: 1.3154...  6.2506 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3039...  Training loss: 1.3074...  6.2604 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3040...  Training loss: 1.3185...  6.2692 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3041...  Training loss: 1.3432...  6.2619 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3042...  Training loss: 1.3234...  6.2147 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3043...  Training loss: 1.3167...  6.2007 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3044...  Training loss: 1.3095...  6.2497 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3045...  Training loss: 1.3239...  6.2402 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3046...  Training loss: 1.3194...  6.2843 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3047...  Training loss: 1.3259...  6.3021 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3048...  Training loss: 1.3113...  6.2346 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3049...  Training loss: 1.3094...  6.2922 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3050...  Training loss: 1.2901...  6.2420 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3051...  Training loss: 1.3081...  6.2362 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3052...  Training loss: 1.3248...  6.2084 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3053...  Training loss: 1.3091...  6.2668 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3054...  Training loss: 1.3215...  6.2024 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3055...  Training loss: 1.2999...  6.2122 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3056...  Training loss: 1.3002...  6.2527 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3057...  Training loss: 1.3065...  6.2463 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3058...  Training loss: 1.2824...  6.2491 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3059...  Training loss: 1.2982...  6.2388 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3060...  Training loss: 1.2774...  6.2789 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3061...  Training loss: 1.3247...  6.2798 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3062...  Training loss: 1.3222...  6.2684 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3063...  Training loss: 1.3699...  6.2260 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3064...  Training loss: 1.3135...  6.2239 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3065...  Training loss: 1.3468...  6.2341 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3066...  Training loss: 1.3030...  6.1895 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3067...  Training loss: 1.3317...  6.1905 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3068...  Training loss: 1.3487...  6.2697 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3069...  Training loss: 1.3414...  6.2192 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3070...  Training loss: 1.3478...  6.2623 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3071...  Training loss: 1.3135...  6.3066 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3072...  Training loss: 1.3699...  6.2732 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3073...  Training loss: 1.3474...  6.2356 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3074...  Training loss: 1.3250...  6.3025 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3075...  Training loss: 1.3368...  6.2919 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3076...  Training loss: 1.2994...  6.3027 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3077...  Training loss: 1.3579...  6.2686 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3078...  Training loss: 1.3188...  6.2294 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3079...  Training loss: 1.3029...  6.2234 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3080...  Training loss: 1.3430...  6.2157 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3081...  Training loss: 1.3321...  6.2140 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3082...  Training loss: 1.3476...  6.3129 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3083...  Training loss: 1.3336...  6.2368 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3084...  Training loss: 1.3310...  6.2996 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3085...  Training loss: 1.3288...  6.2239 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3086...  Training loss: 1.3146...  6.2881 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3087...  Training loss: 1.3213...  6.2499 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3088...  Training loss: 1.3495...  6.3107 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3089...  Training loss: 1.3259...  6.2399 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3090...  Training loss: 1.3501...  6.2919 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3091...  Training loss: 1.3275...  6.2769 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3092...  Training loss: 1.3337...  6.2400 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3093...  Training loss: 1.3464...  6.2979 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3094...  Training loss: 1.3112...  6.3525 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3095...  Training loss: 1.3612...  6.2844 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3096...  Training loss: 1.3241...  6.1999 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3097...  Training loss: 1.3213...  6.2200 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3098...  Training loss: 1.3162...  6.2558 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3099...  Training loss: 1.3619...  6.2863 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3100...  Training loss: 1.3477...  6.2473 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3101...  Training loss: 1.3043...  6.2388 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3102...  Training loss: 1.3295...  6.2304 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3103...  Training loss: 1.3628...  6.3113 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3104...  Training loss: 1.3404...  6.2681 sec/batch\n",
      "Epoch: 15/20...  Training Step: 3105...  Training loss: 1.3494...  6.2540 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3106...  Training loss: 1.4135...  6.2577 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3107...  Training loss: 1.3345...  6.2420 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3108...  Training loss: 1.3399...  6.2184 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3109...  Training loss: 1.3735...  6.2647 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3110...  Training loss: 1.3303...  6.2406 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3111...  Training loss: 1.3310...  6.2601 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3112...  Training loss: 1.3556...  6.2546 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3113...  Training loss: 1.3515...  6.2371 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3114...  Training loss: 1.3568...  6.1723 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3115...  Training loss: 1.3276...  6.2262 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3116...  Training loss: 1.3473...  6.2845 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3117...  Training loss: 1.3034...  6.2929 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3118...  Training loss: 1.3553...  6.3226 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3119...  Training loss: 1.3461...  6.2668 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3120...  Training loss: 1.3408...  6.1968 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3121...  Training loss: 1.2974...  6.2328 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3122...  Training loss: 1.3113...  6.2488 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3123...  Training loss: 1.3291...  6.2055 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3124...  Training loss: 1.3325...  6.2749 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3125...  Training loss: 1.3283...  6.2935 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3126...  Training loss: 1.3197...  6.3165 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3127...  Training loss: 1.3367...  6.2489 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3128...  Training loss: 1.3144...  6.3075 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3129...  Training loss: 1.3195...  6.3032 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3130...  Training loss: 1.3171...  6.3153 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3131...  Training loss: 1.3448...  6.2583 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3132...  Training loss: 1.3188...  6.3367 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3133...  Training loss: 1.3196...  6.3298 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3134...  Training loss: 1.3137...  6.3181 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3135...  Training loss: 1.3353...  6.2723 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3136...  Training loss: 1.2953...  6.2577 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3137...  Training loss: 1.3234...  6.2778 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3138...  Training loss: 1.3440...  6.2465 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3139...  Training loss: 1.3398...  6.2555 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3140...  Training loss: 1.3418...  6.2243 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3141...  Training loss: 1.3143...  6.2564 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3142...  Training loss: 1.3385...  6.2593 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3143...  Training loss: 1.3212...  6.2973 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3144...  Training loss: 1.3096...  6.2778 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3145...  Training loss: 1.3225...  6.3180 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3146...  Training loss: 1.3256...  6.3334 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3147...  Training loss: 1.2977...  6.3156 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3148...  Training loss: 1.3184...  6.3213 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3149...  Training loss: 1.2988...  6.2756 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3150...  Training loss: 1.3032...  6.2702 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3151...  Training loss: 1.3259...  6.2639 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3152...  Training loss: 1.3242...  6.2437 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3153...  Training loss: 1.3143...  6.2567 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3154...  Training loss: 1.3213...  6.2765 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3155...  Training loss: 1.3454...  6.2314 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3156...  Training loss: 1.3195...  6.2880 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3157...  Training loss: 1.3088...  6.2212 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3158...  Training loss: 1.3116...  6.2924 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3159...  Training loss: 1.3212...  6.2845 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3160...  Training loss: 1.2862...  6.2870 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3161...  Training loss: 1.3273...  6.3183 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3162...  Training loss: 1.3165...  6.3044 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3163...  Training loss: 1.3024...  6.2616 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3164...  Training loss: 1.3325...  6.2420 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3165...  Training loss: 1.3305...  6.2660 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3166...  Training loss: 1.3068...  6.3157 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3167...  Training loss: 1.2989...  6.3199 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3168...  Training loss: 1.3333...  6.2654 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3169...  Training loss: 1.3157...  6.3131 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3170...  Training loss: 1.3345...  6.2559 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3171...  Training loss: 1.3776...  6.2554 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3172...  Training loss: 1.3316...  6.3174 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3173...  Training loss: 1.3370...  6.3471 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3174...  Training loss: 1.3389...  6.2627 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3175...  Training loss: 1.3272...  6.3190 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3176...  Training loss: 1.3405...  6.2731 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3177...  Training loss: 1.2967...  6.2513 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3178...  Training loss: 1.3152...  6.2288 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3179...  Training loss: 1.3076...  6.3060 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3180...  Training loss: 1.2938...  6.2362 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3181...  Training loss: 1.3206...  6.2452 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3182...  Training loss: 1.3025...  6.2893 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3183...  Training loss: 1.2884...  6.2714 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3184...  Training loss: 1.3271...  6.2186 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3185...  Training loss: 1.3111...  6.3308 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3186...  Training loss: 1.3171...  6.3227 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3187...  Training loss: 1.3051...  6.2942 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3188...  Training loss: 1.3168...  6.2730 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3189...  Training loss: 1.3194...  6.2514 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3190...  Training loss: 1.3137...  6.2372 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3191...  Training loss: 1.3118...  6.3441 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3192...  Training loss: 1.3221...  6.2808 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3193...  Training loss: 1.3265...  6.2058 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3194...  Training loss: 1.3207...  6.3005 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3195...  Training loss: 1.3217...  6.2086 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3196...  Training loss: 1.2971...  6.2578 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3197...  Training loss: 1.2913...  6.2283 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3198...  Training loss: 1.3177...  6.2617 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3199...  Training loss: 1.2951...  6.3535 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3200...  Training loss: 1.3125...  6.3095 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3201...  Training loss: 1.3007...  6.3285 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3202...  Training loss: 1.3228...  6.3143 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3203...  Training loss: 1.2888...  6.2435 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3204...  Training loss: 1.2781...  6.3133 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3205...  Training loss: 1.3244...  6.2280 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3206...  Training loss: 1.3279...  6.2264 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3207...  Training loss: 1.3266...  6.2894 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3208...  Training loss: 1.2886...  6.2553 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3209...  Training loss: 1.3178...  6.2448 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3210...  Training loss: 1.3052...  6.2492 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3211...  Training loss: 1.2960...  6.2183 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3212...  Training loss: 1.3087...  6.2819 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3213...  Training loss: 1.3329...  6.2335 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3214...  Training loss: 1.3208...  6.3193 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3215...  Training loss: 1.3402...  6.2894 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3216...  Training loss: 1.3435...  6.2954 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3217...  Training loss: 1.2981...  6.3338 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3218...  Training loss: 1.2913...  6.2606 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3219...  Training loss: 1.2946...  6.2785 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3220...  Training loss: 1.3440...  6.2507 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3221...  Training loss: 1.3283...  6.2738 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3222...  Training loss: 1.3394...  6.2401 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3223...  Training loss: 1.3326...  6.2340 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3224...  Training loss: 1.3251...  6.2955 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3225...  Training loss: 1.3105...  6.2746 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3226...  Training loss: 1.2822...  6.2511 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3227...  Training loss: 1.3141...  6.3045 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3228...  Training loss: 1.2946...  6.2479 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3229...  Training loss: 1.2987...  6.3054 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3230...  Training loss: 1.3284...  6.3440 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3231...  Training loss: 1.2887...  6.2588 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3232...  Training loss: 1.3107...  6.3143 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3233...  Training loss: 1.3119...  6.2381 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3234...  Training loss: 1.3081...  6.2341 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3235...  Training loss: 1.3315...  6.2108 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3236...  Training loss: 1.3189...  6.1992 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3237...  Training loss: 1.3133...  6.2254 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3238...  Training loss: 1.3121...  6.2416 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3239...  Training loss: 1.3201...  6.2711 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3240...  Training loss: 1.3064...  6.2230 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3241...  Training loss: 1.3153...  6.3050 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3242...  Training loss: 1.2970...  6.2442 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3243...  Training loss: 1.3204...  6.3231 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3244...  Training loss: 1.3179...  6.2735 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3245...  Training loss: 1.3161...  6.2228 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3246...  Training loss: 1.2903...  6.2899 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3247...  Training loss: 1.3011...  6.2694 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3248...  Training loss: 1.3393...  6.2760 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3249...  Training loss: 1.3050...  6.2772 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3250...  Training loss: 1.3026...  6.2650 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3251...  Training loss: 1.2998...  6.2701 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3252...  Training loss: 1.3158...  6.2546 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3253...  Training loss: 1.3120...  6.3274 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3254...  Training loss: 1.3101...  6.2825 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3255...  Training loss: 1.3046...  6.2583 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3256...  Training loss: 1.2905...  6.3816 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3257...  Training loss: 1.2788...  6.3221 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3258...  Training loss: 1.3022...  6.3549 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3259...  Training loss: 1.2928...  6.2866 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3260...  Training loss: 1.2921...  6.3169 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3261...  Training loss: 1.3049...  6.2827 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3262...  Training loss: 1.2871...  6.3011 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3263...  Training loss: 1.2906...  6.2516 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3264...  Training loss: 1.2814...  6.2547 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3265...  Training loss: 1.2710...  6.2310 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3266...  Training loss: 1.2879...  6.2795 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3267...  Training loss: 1.2778...  6.2059 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3268...  Training loss: 1.3060...  6.2602 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3269...  Training loss: 1.3001...  6.2688 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3270...  Training loss: 1.3498...  6.3278 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3271...  Training loss: 1.3010...  6.2699 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3272...  Training loss: 1.3323...  6.2890 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3273...  Training loss: 1.2918...  6.2375 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3274...  Training loss: 1.3218...  6.2896 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3275...  Training loss: 1.3336...  6.2935 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3276...  Training loss: 1.3269...  6.3004 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3277...  Training loss: 1.3320...  6.2652 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3278...  Training loss: 1.3018...  6.2603 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3279...  Training loss: 1.3542...  6.2659 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3280...  Training loss: 1.3402...  6.2611 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3281...  Training loss: 1.3117...  6.3239 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3282...  Training loss: 1.3286...  6.2620 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3283...  Training loss: 1.2886...  6.2643 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3284...  Training loss: 1.3432...  6.2935 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3285...  Training loss: 1.3133...  6.3210 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3286...  Training loss: 1.2902...  6.2997 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3287...  Training loss: 1.3282...  6.2574 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3288...  Training loss: 1.3293...  6.3294 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3289...  Training loss: 1.3373...  6.2752 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3290...  Training loss: 1.3159...  6.3186 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3291...  Training loss: 1.3221...  6.2282 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3292...  Training loss: 1.3192...  6.2319 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3293...  Training loss: 1.3018...  6.3069 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3294...  Training loss: 1.2977...  6.2440 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3295...  Training loss: 1.3449...  6.2596 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3296...  Training loss: 1.3194...  6.2913 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3297...  Training loss: 1.3248...  6.2519 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3298...  Training loss: 1.3190...  6.2897 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3299...  Training loss: 1.3328...  6.3247 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3300...  Training loss: 1.3201...  6.3696 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3301...  Training loss: 1.3004...  6.2104 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3302...  Training loss: 1.3466...  6.2512 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3303...  Training loss: 1.3055...  6.2678 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3304...  Training loss: 1.3029...  6.2918 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3305...  Training loss: 1.3028...  6.2834 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3306...  Training loss: 1.3485...  6.3770 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3307...  Training loss: 1.3339...  6.2190 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3308...  Training loss: 1.2840...  6.2701 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3309...  Training loss: 1.3214...  6.2617 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3310...  Training loss: 1.3381...  6.3265 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3311...  Training loss: 1.3311...  6.2868 sec/batch\n",
      "Epoch: 16/20...  Training Step: 3312...  Training loss: 1.3366...  6.2973 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3313...  Training loss: 1.4065...  6.3695 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3314...  Training loss: 1.3063...  6.2988 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3315...  Training loss: 1.3237...  6.2664 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3316...  Training loss: 1.3599...  6.2987 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3317...  Training loss: 1.3171...  6.2860 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3318...  Training loss: 1.3173...  6.2491 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3319...  Training loss: 1.3372...  6.3548 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3320...  Training loss: 1.3356...  6.2778 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3321...  Training loss: 1.3391...  6.2684 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3322...  Training loss: 1.3165...  6.2460 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3323...  Training loss: 1.3303...  6.2577 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3324...  Training loss: 1.2991...  6.3222 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3325...  Training loss: 1.3407...  6.3432 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3326...  Training loss: 1.3328...  6.3575 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3327...  Training loss: 1.3156...  6.3692 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3328...  Training loss: 1.2871...  6.2955 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3329...  Training loss: 1.2959...  6.3621 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3330...  Training loss: 1.3099...  6.2841 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3331...  Training loss: 1.3197...  6.3166 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3332...  Training loss: 1.3214...  6.2762 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3333...  Training loss: 1.3043...  6.2830 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3334...  Training loss: 1.3092...  6.2824 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3335...  Training loss: 1.2972...  6.2533 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3336...  Training loss: 1.3022...  6.3213 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3337...  Training loss: 1.3072...  6.3069 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3338...  Training loss: 1.3276...  6.3225 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3339...  Training loss: 1.3141...  6.3179 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3340...  Training loss: 1.3039...  6.3073 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3341...  Training loss: 1.2962...  6.2654 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3342...  Training loss: 1.3157...  6.3028 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3343...  Training loss: 1.2878...  6.2971 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3344...  Training loss: 1.3122...  6.2085 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3345...  Training loss: 1.3253...  6.2770 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3346...  Training loss: 1.3204...  6.2973 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3347...  Training loss: 1.3350...  6.2790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3348...  Training loss: 1.3086...  6.3028 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3349...  Training loss: 1.3276...  6.2412 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3350...  Training loss: 1.3074...  6.2799 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3351...  Training loss: 1.3012...  6.2769 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3352...  Training loss: 1.3137...  6.3144 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3353...  Training loss: 1.3058...  6.2502 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3354...  Training loss: 1.2982...  6.3004 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3355...  Training loss: 1.3103...  6.2789 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3356...  Training loss: 1.2758...  6.2453 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3357...  Training loss: 1.2895...  6.2590 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3358...  Training loss: 1.3011...  6.2928 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3359...  Training loss: 1.3010...  6.2365 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3360...  Training loss: 1.2962...  6.2717 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3361...  Training loss: 1.3144...  6.2279 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3362...  Training loss: 1.3309...  6.2209 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3363...  Training loss: 1.3080...  6.2842 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3364...  Training loss: 1.2930...  6.2508 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3365...  Training loss: 1.3018...  6.2976 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3366...  Training loss: 1.3013...  6.2740 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3367...  Training loss: 1.2704...  6.2770 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3368...  Training loss: 1.3171...  6.3396 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3369...  Training loss: 1.2994...  6.2950 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3370...  Training loss: 1.2927...  6.2787 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3371...  Training loss: 1.3226...  6.2971 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3372...  Training loss: 1.3262...  6.2719 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3373...  Training loss: 1.3032...  6.2345 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3374...  Training loss: 1.2863...  6.2295 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3375...  Training loss: 1.3107...  6.2671 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3376...  Training loss: 1.3093...  6.3537 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3377...  Training loss: 1.3307...  6.2597 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3378...  Training loss: 1.3698...  6.2820 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3379...  Training loss: 1.3159...  6.2839 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3380...  Training loss: 1.3284...  6.2790 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3381...  Training loss: 1.3259...  6.3482 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3382...  Training loss: 1.2989...  6.2529 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3383...  Training loss: 1.3272...  6.2728 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3384...  Training loss: 1.2764...  6.2418 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3385...  Training loss: 1.2983...  6.2249 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3386...  Training loss: 1.2896...  6.2961 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3387...  Training loss: 1.2848...  6.3000 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3388...  Training loss: 1.3051...  6.2075 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3389...  Training loss: 1.2824...  6.2573 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3390...  Training loss: 1.2774...  6.2458 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3391...  Training loss: 1.3207...  6.2265 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3392...  Training loss: 1.2974...  6.2919 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3393...  Training loss: 1.2995...  6.2990 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3394...  Training loss: 1.3080...  6.2628 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3395...  Training loss: 1.3077...  6.2969 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3396...  Training loss: 1.3001...  6.2799 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3397...  Training loss: 1.2993...  6.2689 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3398...  Training loss: 1.3101...  6.2139 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3399...  Training loss: 1.3129...  6.2383 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3400...  Training loss: 1.3216...  6.2110 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3401...  Training loss: 1.3054...  6.2618 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3402...  Training loss: 1.3172...  6.2763 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3403...  Training loss: 1.2888...  6.2476 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3404...  Training loss: 1.2886...  6.2312 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3405...  Training loss: 1.3056...  6.2125 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3406...  Training loss: 1.2762...  6.2153 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3407...  Training loss: 1.3008...  6.2510 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3408...  Training loss: 1.2989...  6.2285 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3409...  Training loss: 1.3169...  6.2659 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3410...  Training loss: 1.2714...  6.2542 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3411...  Training loss: 1.2552...  6.2568 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3412...  Training loss: 1.3119...  6.2314 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3413...  Training loss: 1.3098...  6.2107 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3414...  Training loss: 1.3024...  6.1978 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3415...  Training loss: 1.2714...  6.4045 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3416...  Training loss: 1.3058...  6.2583 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3417...  Training loss: 1.2893...  6.2163 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3418...  Training loss: 1.2838...  6.2401 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3419...  Training loss: 1.3023...  6.2484 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3420...  Training loss: 1.3189...  6.2027 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3421...  Training loss: 1.3057...  6.2725 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3422...  Training loss: 1.3333...  6.2140 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3423...  Training loss: 1.3317...  6.2831 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3424...  Training loss: 1.2872...  6.3129 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3425...  Training loss: 1.2914...  6.1858 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3426...  Training loss: 1.2825...  6.2657 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3427...  Training loss: 1.3243...  6.2099 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3428...  Training loss: 1.3175...  6.1648 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3429...  Training loss: 1.3181...  6.2020 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3430...  Training loss: 1.3177...  6.2068 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3431...  Training loss: 1.3150...  6.2448 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3432...  Training loss: 1.3015...  6.2064 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3433...  Training loss: 1.2758...  6.2057 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3434...  Training loss: 1.3109...  6.3300 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3435...  Training loss: 1.2830...  6.3143 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3436...  Training loss: 1.2923...  6.3130 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3437...  Training loss: 1.3291...  6.3236 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3438...  Training loss: 1.2767...  6.2727 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3439...  Training loss: 1.2953...  6.2110 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3440...  Training loss: 1.3004...  6.2944 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3441...  Training loss: 1.3008...  6.3002 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3442...  Training loss: 1.3223...  6.2688 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3443...  Training loss: 1.3103...  6.2753 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3444...  Training loss: 1.3065...  6.2328 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3445...  Training loss: 1.2911...  6.3212 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3446...  Training loss: 1.2987...  6.2951 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3447...  Training loss: 1.2807...  6.2507 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3448...  Training loss: 1.2939...  6.2874 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3449...  Training loss: 1.2844...  6.2579 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3450...  Training loss: 1.2916...  6.2945 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3451...  Training loss: 1.2993...  6.2825 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3452...  Training loss: 1.2960...  6.2809 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3453...  Training loss: 1.2804...  6.2215 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3454...  Training loss: 1.2997...  6.2563 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3455...  Training loss: 1.3140...  6.2380 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3456...  Training loss: 1.2841...  6.2488 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3457...  Training loss: 1.2935...  6.2010 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3458...  Training loss: 1.2822...  6.2138 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3459...  Training loss: 1.2982...  6.2398 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3460...  Training loss: 1.2947...  6.2337 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3461...  Training loss: 1.3026...  6.2594 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3462...  Training loss: 1.2928...  6.2686 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3463...  Training loss: 1.2734...  6.2654 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3464...  Training loss: 1.2645...  6.3296 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3465...  Training loss: 1.2895...  6.2562 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3466...  Training loss: 1.2834...  6.2417 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3467...  Training loss: 1.2751...  6.2372 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3468...  Training loss: 1.3090...  6.2071 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3469...  Training loss: 1.2696...  6.1829 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3470...  Training loss: 1.2720...  6.1941 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3471...  Training loss: 1.2824...  6.2135 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3472...  Training loss: 1.2680...  6.2905 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3473...  Training loss: 1.2762...  6.2228 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3474...  Training loss: 1.2585...  6.1995 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3475...  Training loss: 1.2923...  6.2548 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3476...  Training loss: 1.2935...  6.2369 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3477...  Training loss: 1.3313...  6.2803 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3478...  Training loss: 1.2844...  6.2150 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3479...  Training loss: 1.3169...  6.2533 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3480...  Training loss: 1.2793...  6.3394 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3481...  Training loss: 1.3194...  6.2021 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3482...  Training loss: 1.3212...  6.2278 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3483...  Training loss: 1.3199...  6.1715 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3484...  Training loss: 1.3194...  6.2563 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3485...  Training loss: 1.2920...  6.2060 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3486...  Training loss: 1.3448...  6.2510 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3487...  Training loss: 1.3303...  6.2310 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3488...  Training loss: 1.3001...  6.2375 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3489...  Training loss: 1.3076...  6.2051 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3490...  Training loss: 1.2804...  6.2206 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3491...  Training loss: 1.3449...  6.2951 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3492...  Training loss: 1.3008...  6.2960 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3493...  Training loss: 1.2807...  6.2441 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3494...  Training loss: 1.3088...  6.2408 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3495...  Training loss: 1.3193...  6.2457 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3496...  Training loss: 1.3322...  6.1863 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3497...  Training loss: 1.3080...  6.2407 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3498...  Training loss: 1.3102...  6.2061 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3499...  Training loss: 1.3028...  6.2455 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3500...  Training loss: 1.2943...  6.1947 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3501...  Training loss: 1.2937...  6.2405 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3502...  Training loss: 1.3306...  6.2590 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3503...  Training loss: 1.3075...  6.2892 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3504...  Training loss: 1.3147...  6.2595 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3505...  Training loss: 1.3037...  6.2891 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3506...  Training loss: 1.3177...  6.2479 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3507...  Training loss: 1.3179...  6.2542 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3508...  Training loss: 1.2904...  6.2688 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3509...  Training loss: 1.3330...  6.2752 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3510...  Training loss: 1.3016...  6.2650 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3511...  Training loss: 1.2985...  6.2616 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3512...  Training loss: 1.2922...  6.2467 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3513...  Training loss: 1.3369...  6.1584 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3514...  Training loss: 1.3268...  6.2250 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3515...  Training loss: 1.2822...  6.2983 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3516...  Training loss: 1.3032...  6.2191 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3517...  Training loss: 1.3323...  6.2191 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3518...  Training loss: 1.3115...  6.2323 sec/batch\n",
      "Epoch: 17/20...  Training Step: 3519...  Training loss: 1.3157...  6.2850 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3520...  Training loss: 1.3773...  6.2658 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3521...  Training loss: 1.3066...  6.2403 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3522...  Training loss: 1.2961...  6.2342 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3523...  Training loss: 1.3435...  6.3079 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3524...  Training loss: 1.3083...  6.2420 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3525...  Training loss: 1.3061...  6.2382 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3526...  Training loss: 1.3253...  6.2477 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3527...  Training loss: 1.3260...  6.1823 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3528...  Training loss: 1.3209...  6.2296 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3529...  Training loss: 1.3202...  6.2753 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3530...  Training loss: 1.3171...  6.3060 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3531...  Training loss: 1.2845...  6.2468 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3532...  Training loss: 1.3342...  6.2312 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3533...  Training loss: 1.3116...  6.2298 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3534...  Training loss: 1.3051...  6.2223 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3535...  Training loss: 1.2692...  6.2686 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3536...  Training loss: 1.2848...  6.2837 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3537...  Training loss: 1.3051...  6.2018 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3538...  Training loss: 1.3088...  6.2147 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3539...  Training loss: 1.2998...  6.2941 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3540...  Training loss: 1.2824...  6.2204 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3541...  Training loss: 1.2996...  6.2412 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3542...  Training loss: 1.2936...  6.2692 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3543...  Training loss: 1.2999...  6.2196 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3544...  Training loss: 1.2936...  6.1959 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3545...  Training loss: 1.3215...  6.2589 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3546...  Training loss: 1.3024...  6.1920 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3547...  Training loss: 1.2977...  6.2915 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3548...  Training loss: 1.2874...  6.2302 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3549...  Training loss: 1.3099...  6.2061 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3550...  Training loss: 1.2678...  6.2410 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3551...  Training loss: 1.2987...  6.2217 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3552...  Training loss: 1.3194...  6.1953 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3553...  Training loss: 1.3147...  6.2491 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3554...  Training loss: 1.3108...  6.2659 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3555...  Training loss: 1.2845...  6.2267 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3556...  Training loss: 1.3003...  6.1830 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3557...  Training loss: 1.2892...  6.2439 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3558...  Training loss: 1.2927...  6.2130 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3559...  Training loss: 1.2950...  6.2154 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3560...  Training loss: 1.2928...  6.2323 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3561...  Training loss: 1.2848...  6.2652 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3562...  Training loss: 1.2937...  6.2287 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3563...  Training loss: 1.2635...  6.2682 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3564...  Training loss: 1.2772...  6.2678 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3565...  Training loss: 1.2925...  6.2635 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3566...  Training loss: 1.2934...  6.1838 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3567...  Training loss: 1.2895...  6.2131 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3568...  Training loss: 1.3047...  6.2748 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3569...  Training loss: 1.3131...  6.2936 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3570...  Training loss: 1.2995...  6.2579 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3571...  Training loss: 1.2830...  6.2620 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3572...  Training loss: 1.2869...  6.1873 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3573...  Training loss: 1.2901...  6.2052 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3574...  Training loss: 1.2646...  6.2385 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3575...  Training loss: 1.3034...  6.3011 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3576...  Training loss: 1.2918...  6.2179 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3577...  Training loss: 1.2812...  6.2946 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3578...  Training loss: 1.3125...  6.2876 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3579...  Training loss: 1.3113...  6.2005 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3580...  Training loss: 1.2771...  6.2284 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3581...  Training loss: 1.2668...  6.2848 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3582...  Training loss: 1.3128...  6.2430 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3583...  Training loss: 1.2884...  6.2402 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3584...  Training loss: 1.3061...  6.2731 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3585...  Training loss: 1.3555...  6.2672 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3586...  Training loss: 1.2962...  6.2335 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3587...  Training loss: 1.3207...  6.2423 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3588...  Training loss: 1.3050...  6.2116 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3589...  Training loss: 1.2988...  6.2710 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3590...  Training loss: 1.3178...  6.2709 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3591...  Training loss: 1.2709...  6.2381 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3592...  Training loss: 1.2880...  6.2830 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3593...  Training loss: 1.2793...  6.2161 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3594...  Training loss: 1.2651...  6.2424 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3595...  Training loss: 1.2983...  6.1991 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3596...  Training loss: 1.2731...  6.2453 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3597...  Training loss: 1.2678...  6.2152 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3598...  Training loss: 1.3125...  6.1761 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3599...  Training loss: 1.2804...  6.2233 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3600...  Training loss: 1.2891...  6.2570 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3601...  Training loss: 1.2911...  6.2357 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3602...  Training loss: 1.2909...  6.2303 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3603...  Training loss: 1.2878...  6.2869 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3604...  Training loss: 1.2750...  6.3344 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3605...  Training loss: 1.2870...  6.2501 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3606...  Training loss: 1.2973...  6.3180 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3607...  Training loss: 1.3009...  6.2848 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3608...  Training loss: 1.2870...  6.2317 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3609...  Training loss: 1.3002...  6.2327 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3610...  Training loss: 1.2707...  6.2490 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3611...  Training loss: 1.2677...  6.2400 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3612...  Training loss: 1.2976...  6.2493 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3613...  Training loss: 1.2640...  6.1795 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3614...  Training loss: 1.2799...  6.1731 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3615...  Training loss: 1.2774...  6.2064 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3616...  Training loss: 1.3045...  6.2719 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3617...  Training loss: 1.2670...  6.2805 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3618...  Training loss: 1.2432...  6.2493 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3619...  Training loss: 1.2960...  6.2868 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3620...  Training loss: 1.3034...  6.2709 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3621...  Training loss: 1.2999...  6.2572 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3622...  Training loss: 1.2621...  6.1750 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3623...  Training loss: 1.2980...  6.2331 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3624...  Training loss: 1.2719...  6.2544 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3625...  Training loss: 1.2729...  6.2000 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3626...  Training loss: 1.2758...  6.2035 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3627...  Training loss: 1.3040...  6.2452 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3628...  Training loss: 1.2894...  6.2127 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3629...  Training loss: 1.3195...  6.2337 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3630...  Training loss: 1.3222...  6.2407 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3631...  Training loss: 1.2801...  6.2134 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3632...  Training loss: 1.2707...  6.2276 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3633...  Training loss: 1.2769...  6.2839 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3634...  Training loss: 1.3031...  6.2225 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3635...  Training loss: 1.3128...  6.2182 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3636...  Training loss: 1.3108...  6.2457 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3637...  Training loss: 1.3116...  6.1984 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3638...  Training loss: 1.3055...  6.2667 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3639...  Training loss: 1.2876...  6.2835 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3640...  Training loss: 1.2611...  6.2270 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3641...  Training loss: 1.2999...  6.2004 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3642...  Training loss: 1.2610...  6.2350 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3643...  Training loss: 1.2768...  6.2521 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3644...  Training loss: 1.3043...  6.2543 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3645...  Training loss: 1.2673...  6.2219 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3646...  Training loss: 1.2842...  6.2741 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3647...  Training loss: 1.2838...  6.2826 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3648...  Training loss: 1.2940...  6.2264 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3649...  Training loss: 1.3037...  6.2132 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3650...  Training loss: 1.2914...  6.2365 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3651...  Training loss: 1.2846...  6.2291 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3652...  Training loss: 1.2908...  6.2368 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3653...  Training loss: 1.2985...  6.1822 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3654...  Training loss: 1.2837...  6.1887 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3655...  Training loss: 1.2812...  6.2475 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3656...  Training loss: 1.2732...  6.2759 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3657...  Training loss: 1.2888...  6.1957 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3658...  Training loss: 1.2833...  6.2900 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3659...  Training loss: 1.2895...  6.2600 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3660...  Training loss: 1.2775...  6.2689 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3661...  Training loss: 1.2870...  6.2683 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3662...  Training loss: 1.2977...  6.2464 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3663...  Training loss: 1.2854...  6.2035 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3664...  Training loss: 1.2765...  6.2496 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3665...  Training loss: 1.2707...  6.1823 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3666...  Training loss: 1.2841...  6.3045 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3667...  Training loss: 1.2847...  6.2411 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3668...  Training loss: 1.2810...  6.2691 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3669...  Training loss: 1.2817...  6.2217 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3670...  Training loss: 1.2656...  6.2390 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3671...  Training loss: 1.2578...  6.2597 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3672...  Training loss: 1.2681...  6.2376 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3673...  Training loss: 1.2724...  6.3215 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3674...  Training loss: 1.2639...  6.2686 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3675...  Training loss: 1.2953...  6.2398 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3676...  Training loss: 1.2636...  6.2206 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3677...  Training loss: 1.2596...  6.2454 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3678...  Training loss: 1.2661...  6.2432 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3679...  Training loss: 1.2581...  6.1946 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3680...  Training loss: 1.2567...  6.2561 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3681...  Training loss: 1.2406...  6.2271 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3682...  Training loss: 1.2798...  6.2025 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3683...  Training loss: 1.2779...  6.2413 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3684...  Training loss: 1.3235...  6.2404 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3685...  Training loss: 1.2727...  6.1858 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3686...  Training loss: 1.3088...  6.2932 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3687...  Training loss: 1.2635...  6.2961 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3688...  Training loss: 1.2990...  6.2648 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3689...  Training loss: 1.3072...  6.2805 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3690...  Training loss: 1.3019...  6.2260 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3691...  Training loss: 1.3065...  6.2403 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3692...  Training loss: 1.2835...  6.2280 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3693...  Training loss: 1.3386...  6.2275 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3694...  Training loss: 1.3047...  6.2857 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3695...  Training loss: 1.2889...  6.2594 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3696...  Training loss: 1.3001...  6.2657 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3697...  Training loss: 1.2611...  6.2520 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3698...  Training loss: 1.3236...  6.1733 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3699...  Training loss: 1.2901...  6.2808 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3700...  Training loss: 1.2650...  6.2999 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3701...  Training loss: 1.3034...  6.3148 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3702...  Training loss: 1.3167...  6.2891 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3703...  Training loss: 1.3217...  6.3391 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3704...  Training loss: 1.2887...  6.2544 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3705...  Training loss: 1.2893...  6.2499 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3706...  Training loss: 1.3048...  6.2493 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3707...  Training loss: 1.2833...  6.1865 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3708...  Training loss: 1.2839...  6.2143 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3709...  Training loss: 1.3163...  6.2563 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3710...  Training loss: 1.2970...  6.1886 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3711...  Training loss: 1.2977...  6.2742 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3712...  Training loss: 1.2981...  6.2540 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3713...  Training loss: 1.3121...  6.2615 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3714...  Training loss: 1.3115...  6.2695 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3715...  Training loss: 1.2807...  6.2687 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3716...  Training loss: 1.3223...  6.3036 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3717...  Training loss: 1.2909...  6.2667 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3718...  Training loss: 1.2858...  6.2296 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3719...  Training loss: 1.2796...  6.2297 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3720...  Training loss: 1.3219...  6.2424 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3721...  Training loss: 1.3174...  6.2407 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3722...  Training loss: 1.2712...  6.2393 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3723...  Training loss: 1.2936...  6.2111 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3724...  Training loss: 1.3230...  6.2164 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3725...  Training loss: 1.3044...  6.1947 sec/batch\n",
      "Epoch: 18/20...  Training Step: 3726...  Training loss: 1.3055...  6.2313 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3727...  Training loss: 1.3702...  6.2023 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3728...  Training loss: 1.2941...  6.2637 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3729...  Training loss: 1.3020...  6.2411 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3730...  Training loss: 1.3280...  6.2528 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3731...  Training loss: 1.3043...  6.3148 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3732...  Training loss: 1.3043...  6.2420 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3733...  Training loss: 1.3159...  6.2162 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3734...  Training loss: 1.3165...  6.2283 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3735...  Training loss: 1.3190...  6.1928 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3736...  Training loss: 1.2971...  6.2119 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3737...  Training loss: 1.3039...  6.2246 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3738...  Training loss: 1.2777...  6.2400 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3739...  Training loss: 1.3154...  6.2698 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3740...  Training loss: 1.3017...  6.2170 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3741...  Training loss: 1.2919...  6.2829 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3742...  Training loss: 1.2671...  6.2672 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3743...  Training loss: 1.2661...  6.2380 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3744...  Training loss: 1.2977...  6.2339 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3745...  Training loss: 1.2959...  6.2989 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3746...  Training loss: 1.2887...  6.2682 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3747...  Training loss: 1.2882...  6.2875 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3748...  Training loss: 1.2882...  6.2328 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3749...  Training loss: 1.2757...  6.2498 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3750...  Training loss: 1.2812...  6.2496 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3751...  Training loss: 1.2793...  6.2384 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3752...  Training loss: 1.2997...  6.1903 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3753...  Training loss: 1.2956...  6.2042 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3754...  Training loss: 1.2916...  6.2730 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3755...  Training loss: 1.2802...  6.2171 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3756...  Training loss: 1.2939...  6.2374 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3757...  Training loss: 1.2607...  6.2646 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3758...  Training loss: 1.2876...  6.2565 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3759...  Training loss: 1.3135...  6.2762 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3760...  Training loss: 1.3012...  6.2403 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3761...  Training loss: 1.2989...  6.2166 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3762...  Training loss: 1.2711...  6.1984 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3763...  Training loss: 1.2969...  6.2184 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3764...  Training loss: 1.2786...  6.2182 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3765...  Training loss: 1.2792...  6.2310 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3766...  Training loss: 1.2922...  6.2434 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3767...  Training loss: 1.2865...  6.2748 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3768...  Training loss: 1.2705...  6.2860 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3769...  Training loss: 1.2739...  6.2064 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3770...  Training loss: 1.2495...  6.2269 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3771...  Training loss: 1.2627...  6.2907 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3772...  Training loss: 1.2732...  6.2709 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3773...  Training loss: 1.2817...  6.2161 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3774...  Training loss: 1.2754...  6.2828 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3775...  Training loss: 1.2796...  6.2071 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3776...  Training loss: 1.2974...  6.2071 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3777...  Training loss: 1.2851...  6.2628 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3778...  Training loss: 1.2742...  6.1968 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3779...  Training loss: 1.2740...  6.2425 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3780...  Training loss: 1.2792...  6.2515 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3781...  Training loss: 1.2534...  6.2402 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3782...  Training loss: 1.2965...  6.2656 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3783...  Training loss: 1.2861...  6.2154 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3784...  Training loss: 1.2758...  6.2978 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3785...  Training loss: 1.3044...  6.2615 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3786...  Training loss: 1.2980...  6.2511 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3787...  Training loss: 1.2777...  6.2388 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3788...  Training loss: 1.2653...  6.2251 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3789...  Training loss: 1.3055...  6.2477 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3790...  Training loss: 1.2795...  6.2585 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3791...  Training loss: 1.2912...  6.2515 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3792...  Training loss: 1.3431...  6.2636 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3793...  Training loss: 1.2855...  6.2043 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3794...  Training loss: 1.3092...  6.1993 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3795...  Training loss: 1.3071...  6.2628 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3796...  Training loss: 1.2946...  6.3003 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3797...  Training loss: 1.3086...  6.2704 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3798...  Training loss: 1.2558...  6.2769 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3799...  Training loss: 1.2774...  6.3428 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3800...  Training loss: 1.2754...  6.3351 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3801...  Training loss: 1.2643...  6.1893 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3802...  Training loss: 1.2844...  6.2639 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3803...  Training loss: 1.2613...  6.2268 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3804...  Training loss: 1.2533...  6.2142 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3805...  Training loss: 1.2952...  6.2849 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3806...  Training loss: 1.2772...  6.2600 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3807...  Training loss: 1.2781...  6.2173 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3808...  Training loss: 1.2690...  6.2546 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3809...  Training loss: 1.2832...  6.2220 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3810...  Training loss: 1.2937...  6.2476 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3811...  Training loss: 1.2705...  6.2818 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3812...  Training loss: 1.2891...  6.1949 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3813...  Training loss: 1.2814...  6.2650 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3814...  Training loss: 1.2925...  6.2683 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3815...  Training loss: 1.2775...  6.2383 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3816...  Training loss: 1.2837...  6.2230 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3817...  Training loss: 1.2658...  6.1730 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3818...  Training loss: 1.2583...  6.2198 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3819...  Training loss: 1.2880...  6.2895 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3820...  Training loss: 1.2594...  6.2375 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3821...  Training loss: 1.2714...  6.2281 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3822...  Training loss: 1.2683...  6.2620 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3823...  Training loss: 1.2953...  6.2219 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3824...  Training loss: 1.2485...  6.2416 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3825...  Training loss: 1.2411...  6.1928 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3826...  Training loss: 1.2909...  6.2896 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3827...  Training loss: 1.2884...  6.2587 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3828...  Training loss: 1.2927...  6.3179 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3829...  Training loss: 1.2502...  6.2229 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3830...  Training loss: 1.2840...  6.1906 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3831...  Training loss: 1.2662...  6.2431 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3832...  Training loss: 1.2653...  6.2396 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3833...  Training loss: 1.2822...  6.2293 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3834...  Training loss: 1.2846...  6.2369 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3835...  Training loss: 1.2858...  6.2173 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3836...  Training loss: 1.3044...  6.2613 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3837...  Training loss: 1.3192...  6.2261 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3838...  Training loss: 1.2608...  6.1967 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3839...  Training loss: 1.2683...  6.2570 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3840...  Training loss: 1.2596...  6.2645 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3841...  Training loss: 1.2999...  6.2473 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3842...  Training loss: 1.3000...  6.2391 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3843...  Training loss: 1.2946...  6.2668 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3844...  Training loss: 1.2908...  6.2221 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3845...  Training loss: 1.2962...  6.2233 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3846...  Training loss: 1.2763...  6.2523 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3847...  Training loss: 1.2427...  6.2680 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3848...  Training loss: 1.2822...  6.2168 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3849...  Training loss: 1.2653...  6.2644 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3850...  Training loss: 1.2658...  6.2856 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3851...  Training loss: 1.2948...  6.2969 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3852...  Training loss: 1.2472...  6.2474 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3853...  Training loss: 1.2798...  6.1857 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3854...  Training loss: 1.2669...  6.2179 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3855...  Training loss: 1.2792...  6.2601 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3856...  Training loss: 1.2906...  6.3150 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3857...  Training loss: 1.2725...  6.2571 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3858...  Training loss: 1.2828...  6.2001 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3859...  Training loss: 1.2814...  6.1902 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3860...  Training loss: 1.2816...  6.2077 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3861...  Training loss: 1.2631...  6.2892 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3862...  Training loss: 1.2806...  6.2915 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3863...  Training loss: 1.2639...  6.2116 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3864...  Training loss: 1.2704...  6.2924 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3865...  Training loss: 1.2798...  6.2534 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3866...  Training loss: 1.2738...  6.2321 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3867...  Training loss: 1.2612...  6.2360 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3868...  Training loss: 1.2714...  6.2455 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3869...  Training loss: 1.2973...  6.2687 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3870...  Training loss: 1.2704...  6.3154 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3871...  Training loss: 1.2655...  6.2635 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3872...  Training loss: 1.2460...  6.2307 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3873...  Training loss: 1.2779...  6.2559 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3874...  Training loss: 1.2620...  6.2456 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3875...  Training loss: 1.2722...  6.2738 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3876...  Training loss: 1.2772...  6.2480 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3877...  Training loss: 1.2510...  6.2527 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3878...  Training loss: 1.2428...  6.2482 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3879...  Training loss: 1.2593...  6.2371 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3880...  Training loss: 1.2654...  6.2194 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3881...  Training loss: 1.2513...  6.2672 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3882...  Training loss: 1.2776...  6.2905 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3883...  Training loss: 1.2522...  6.2755 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3884...  Training loss: 1.2504...  6.3005 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3885...  Training loss: 1.2533...  6.2118 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3886...  Training loss: 1.2367...  6.2409 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3887...  Training loss: 1.2551...  6.1993 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3888...  Training loss: 1.2410...  6.2019 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3889...  Training loss: 1.2760...  6.2241 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3890...  Training loss: 1.2756...  6.2302 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3891...  Training loss: 1.3138...  6.1784 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3892...  Training loss: 1.2623...  6.2156 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3893...  Training loss: 1.2981...  6.2412 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3894...  Training loss: 1.2627...  6.2637 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3895...  Training loss: 1.2896...  6.2686 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3896...  Training loss: 1.3036...  6.2044 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3897...  Training loss: 1.3033...  6.2357 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3898...  Training loss: 1.2882...  6.3150 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3899...  Training loss: 1.2670...  6.2752 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3900...  Training loss: 1.3315...  6.2558 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3901...  Training loss: 1.3022...  6.2444 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3902...  Training loss: 1.2753...  6.2669 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3903...  Training loss: 1.2848...  6.1713 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3904...  Training loss: 1.2513...  6.2464 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3905...  Training loss: 1.3099...  6.2672 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3906...  Training loss: 1.2744...  6.2671 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3907...  Training loss: 1.2579...  6.2114 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3908...  Training loss: 1.2865...  6.2835 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3909...  Training loss: 1.2848...  6.2529 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3910...  Training loss: 1.3124...  6.2952 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3911...  Training loss: 1.2854...  6.2411 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3912...  Training loss: 1.2736...  6.2824 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3913...  Training loss: 1.2883...  6.2213 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3914...  Training loss: 1.2807...  6.2686 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3915...  Training loss: 1.2745...  6.2806 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3916...  Training loss: 1.3046...  6.2444 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3917...  Training loss: 1.2805...  6.2347 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3918...  Training loss: 1.2919...  6.2007 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3919...  Training loss: 1.2954...  6.2285 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3920...  Training loss: 1.2936...  6.2184 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3921...  Training loss: 1.2948...  6.2077 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3922...  Training loss: 1.2734...  6.2060 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3923...  Training loss: 1.3132...  6.3120 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3924...  Training loss: 1.2756...  6.2782 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3925...  Training loss: 1.2681...  6.2409 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3926...  Training loss: 1.2658...  6.3014 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3927...  Training loss: 1.3194...  6.1867 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3928...  Training loss: 1.3045...  6.2898 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3929...  Training loss: 1.2579...  6.2301 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3930...  Training loss: 1.2785...  6.2628 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3931...  Training loss: 1.3105...  6.2409 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3932...  Training loss: 1.2924...  6.2269 sec/batch\n",
      "Epoch: 19/20...  Training Step: 3933...  Training loss: 1.2995...  6.2438 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3934...  Training loss: 1.3768...  6.2366 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3935...  Training loss: 1.2718...  6.2564 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3936...  Training loss: 1.2908...  6.2627 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3937...  Training loss: 1.3206...  6.2469 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3938...  Training loss: 1.2871...  6.2089 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3939...  Training loss: 1.2930...  6.2913 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3940...  Training loss: 1.3010...  6.2696 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3941...  Training loss: 1.3002...  6.2637 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3942...  Training loss: 1.2955...  6.2057 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3943...  Training loss: 1.2942...  6.2669 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3944...  Training loss: 1.2969...  6.2237 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3945...  Training loss: 1.2642...  6.1700 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3946...  Training loss: 1.3038...  6.2017 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3947...  Training loss: 1.2862...  6.2380 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3948...  Training loss: 1.2864...  6.2560 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3949...  Training loss: 1.2471...  6.2157 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3950...  Training loss: 1.2663...  6.2358 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3951...  Training loss: 1.2855...  6.3643 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3952...  Training loss: 1.2971...  6.2808 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3953...  Training loss: 1.2740...  6.2662 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3954...  Training loss: 1.2827...  6.2874 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3955...  Training loss: 1.2726...  6.1960 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3956...  Training loss: 1.2731...  6.2220 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3957...  Training loss: 1.2664...  6.1878 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3958...  Training loss: 1.2726...  6.2356 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3959...  Training loss: 1.3004...  6.2360 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3960...  Training loss: 1.2815...  6.3008 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3961...  Training loss: 1.2767...  6.2069 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3962...  Training loss: 1.2683...  6.2256 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3963...  Training loss: 1.2798...  6.2189 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3964...  Training loss: 1.2556...  6.2312 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3965...  Training loss: 1.2731...  6.2547 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3966...  Training loss: 1.3059...  6.2931 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3967...  Training loss: 1.2946...  6.2657 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3968...  Training loss: 1.2991...  6.2323 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3969...  Training loss: 1.2672...  6.2118 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3970...  Training loss: 1.2855...  6.2712 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3971...  Training loss: 1.2700...  6.2677 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3972...  Training loss: 1.2685...  6.2585 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3973...  Training loss: 1.2798...  6.2339 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3974...  Training loss: 1.2811...  6.2273 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3975...  Training loss: 1.2620...  6.2267 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3976...  Training loss: 1.2738...  6.2270 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3977...  Training loss: 1.2422...  6.2552 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3978...  Training loss: 1.2538...  6.2513 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3979...  Training loss: 1.2693...  6.3346 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3980...  Training loss: 1.2712...  6.3107 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3981...  Training loss: 1.2690...  6.3083 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3982...  Training loss: 1.2695...  6.2455 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3983...  Training loss: 1.2904...  6.2007 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3984...  Training loss: 1.2729...  6.2073 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3985...  Training loss: 1.2637...  6.2442 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3986...  Training loss: 1.2594...  6.2498 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3987...  Training loss: 1.2644...  6.2200 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3988...  Training loss: 1.2390...  6.2954 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3989...  Training loss: 1.2826...  6.2885 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3990...  Training loss: 1.2685...  6.2393 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3991...  Training loss: 1.2526...  6.2207 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3992...  Training loss: 1.2815...  6.2093 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3993...  Training loss: 1.2833...  6.3003 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3994...  Training loss: 1.2575...  6.2591 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3995...  Training loss: 1.2476...  6.2815 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3996...  Training loss: 1.2928...  6.2419 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3997...  Training loss: 1.2609...  6.2438 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3998...  Training loss: 1.2902...  6.2480 sec/batch\n",
      "Epoch: 20/20...  Training Step: 3999...  Training loss: 1.3358...  6.2079 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4000...  Training loss: 1.2769...  6.2833 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4001...  Training loss: 1.2927...  6.3145 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4002...  Training loss: 1.2943...  6.2309 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4003...  Training loss: 1.2780...  6.2374 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4004...  Training loss: 1.2992...  6.1919 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4005...  Training loss: 1.2512...  6.2515 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4006...  Training loss: 1.2606...  6.1856 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4007...  Training loss: 1.2667...  6.2543 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4008...  Training loss: 1.2573...  6.2581 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4009...  Training loss: 1.2788...  6.2954 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4010...  Training loss: 1.2549...  6.2000 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4011...  Training loss: 1.2430...  6.2660 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4012...  Training loss: 1.2854...  6.2515 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4013...  Training loss: 1.2650...  6.2147 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4014...  Training loss: 1.2644...  6.2117 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4015...  Training loss: 1.2634...  6.2146 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4016...  Training loss: 1.2670...  6.2663 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4017...  Training loss: 1.2776...  6.2953 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4018...  Training loss: 1.2616...  6.2522 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4019...  Training loss: 1.2708...  6.2888 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4020...  Training loss: 1.2767...  6.2292 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4021...  Training loss: 1.2797...  6.2437 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4022...  Training loss: 1.2708...  6.3056 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4023...  Training loss: 1.2723...  6.2769 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4024...  Training loss: 1.2592...  6.2122 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4025...  Training loss: 1.2514...  6.2650 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4026...  Training loss: 1.2777...  6.2326 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4027...  Training loss: 1.2490...  6.2151 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4028...  Training loss: 1.2670...  6.2545 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4029...  Training loss: 1.2553...  6.3192 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4030...  Training loss: 1.2831...  6.2298 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4031...  Training loss: 1.2400...  6.2437 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4032...  Training loss: 1.2399...  6.2664 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4033...  Training loss: 1.2856...  6.2540 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4034...  Training loss: 1.2772...  6.3225 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4035...  Training loss: 1.2735...  6.2444 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4036...  Training loss: 1.2456...  6.3406 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4037...  Training loss: 1.2612...  6.2388 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4038...  Training loss: 1.2478...  6.2429 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4039...  Training loss: 1.2563...  6.2101 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4040...  Training loss: 1.2595...  6.2207 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4041...  Training loss: 1.2846...  6.2677 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4042...  Training loss: 1.2773...  6.2225 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4043...  Training loss: 1.2944...  6.2659 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4044...  Training loss: 1.3016...  6.2510 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4045...  Training loss: 1.2429...  6.2142 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4046...  Training loss: 1.2455...  6.2056 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4047...  Training loss: 1.2498...  6.2312 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4048...  Training loss: 1.2839...  6.2794 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4049...  Training loss: 1.2789...  6.2490 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4050...  Training loss: 1.2833...  6.2930 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4051...  Training loss: 1.2804...  6.2424 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4052...  Training loss: 1.2821...  6.2088 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4053...  Training loss: 1.2634...  6.1831 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4054...  Training loss: 1.2437...  6.2551 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4055...  Training loss: 1.2775...  6.2802 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4056...  Training loss: 1.2502...  6.2631 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4057...  Training loss: 1.2560...  6.2996 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4058...  Training loss: 1.2816...  6.2779 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4059...  Training loss: 1.2446...  6.2938 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4060...  Training loss: 1.2601...  6.2543 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4061...  Training loss: 1.2659...  6.2694 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4062...  Training loss: 1.2775...  6.2789 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4063...  Training loss: 1.2843...  6.2575 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4064...  Training loss: 1.2768...  6.3531 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4065...  Training loss: 1.2730...  6.2344 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4066...  Training loss: 1.2651...  6.1904 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4067...  Training loss: 1.2737...  6.2166 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4068...  Training loss: 1.2530...  6.2168 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4069...  Training loss: 1.2595...  6.2711 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4070...  Training loss: 1.2452...  6.2515 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4071...  Training loss: 1.2539...  6.2971 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4072...  Training loss: 1.2580...  6.1870 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4073...  Training loss: 1.2632...  6.2463 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4074...  Training loss: 1.2568...  6.2702 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4075...  Training loss: 1.2657...  6.3100 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4076...  Training loss: 1.2847...  6.2387 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4077...  Training loss: 1.2511...  6.2549 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4078...  Training loss: 1.2684...  6.3047 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4079...  Training loss: 1.2468...  6.2938 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4080...  Training loss: 1.2594...  6.2425 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4081...  Training loss: 1.2652...  6.2651 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4082...  Training loss: 1.2711...  6.2528 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4083...  Training loss: 1.2577...  6.2496 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4084...  Training loss: 1.2416...  6.2125 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4085...  Training loss: 1.2401...  6.2059 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4086...  Training loss: 1.2520...  6.2319 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4087...  Training loss: 1.2519...  6.1894 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4088...  Training loss: 1.2485...  6.2277 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4089...  Training loss: 1.2786...  6.2933 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4090...  Training loss: 1.2392...  6.3514 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4091...  Training loss: 1.2368...  6.2848 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4092...  Training loss: 1.2452...  6.2828 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4093...  Training loss: 1.2228...  6.2953 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4094...  Training loss: 1.2384...  6.1989 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4095...  Training loss: 1.2201...  6.2137 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4096...  Training loss: 1.2565...  6.2526 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4097...  Training loss: 1.2680...  6.2068 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4098...  Training loss: 1.3014...  6.2206 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4099...  Training loss: 1.2579...  6.2624 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4100...  Training loss: 1.2850...  6.2652 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4101...  Training loss: 1.2519...  6.2483 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4102...  Training loss: 1.2732...  6.1796 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4103...  Training loss: 1.2879...  6.2704 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4104...  Training loss: 1.2873...  6.2628 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4105...  Training loss: 1.2888...  6.3034 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4106...  Training loss: 1.2526...  6.3381 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4107...  Training loss: 1.3136...  6.2407 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4108...  Training loss: 1.2951...  6.2563 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4109...  Training loss: 1.2700...  6.2126 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4110...  Training loss: 1.2766...  6.2317 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4111...  Training loss: 1.2444...  6.2690 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4112...  Training loss: 1.3040...  6.2328 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4113...  Training loss: 1.2735...  6.2301 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4114...  Training loss: 1.2494...  6.2450 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4115...  Training loss: 1.2721...  6.2177 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4116...  Training loss: 1.2873...  6.2369 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4117...  Training loss: 1.3064...  6.2567 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4118...  Training loss: 1.2811...  6.2698 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4119...  Training loss: 1.2710...  6.2766 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4120...  Training loss: 1.2687...  6.2505 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4121...  Training loss: 1.2626...  6.2519 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4122...  Training loss: 1.2680...  6.2330 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4123...  Training loss: 1.2975...  6.2383 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4124...  Training loss: 1.2638...  6.2343 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4125...  Training loss: 1.2780...  6.2582 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4126...  Training loss: 1.2746...  6.2411 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4127...  Training loss: 1.2823...  6.2232 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4128...  Training loss: 1.2831...  6.2354 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4129...  Training loss: 1.2618...  6.1937 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4130...  Training loss: 1.3028...  6.2725 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4131...  Training loss: 1.2609...  6.2637 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4132...  Training loss: 1.2661...  6.2830 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4133...  Training loss: 1.2644...  6.2827 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4134...  Training loss: 1.3097...  6.3292 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4135...  Training loss: 1.2886...  6.2699 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4136...  Training loss: 1.2446...  6.2548 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4137...  Training loss: 1.2751...  6.2327 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4138...  Training loss: 1.2973...  6.2547 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4139...  Training loss: 1.2835...  6.2073 sec/batch\n",
      "Epoch: 20/20...  Training Step: 4140...  Training loss: 1.2892...  6.2920 sec/batch\n"
     ]
    }
   ],
   "source": [
    "epochs = 20\n",
    "# Save every N iterations\n",
    "save_every_n = 200\n",
    "\n",
    "model = CharRNN(len(vocab), batch_size=batch_size, num_steps=num_steps,\n",
    "                lstm_size=lstm_size, num_layers=num_layers, \n",
    "                learning_rate=learning_rate)\n",
    "\n",
    "saver = tf.train.Saver(max_to_keep=100)\n",
    "with tf.Session() as sess:\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    \n",
    "    # Use the line below to load a checkpoint and resume training\n",
    "    #saver.restore(sess, 'checkpoints/______.ckpt')\n",
    "    counter = 0\n",
    "    for e in range(epochs):\n",
    "        # Train network\n",
    "        new_state = sess.run(model.initial_state)\n",
    "        loss = 0\n",
    "        for x, y in get_batches(encoded, batch_size, num_steps):\n",
    "            counter += 1\n",
    "            start = time.time()\n",
    "            feed = {model.inputs: x,\n",
    "                    model.targets: y,\n",
    "                    model.keep_prob: keep_prob,\n",
    "                    model.initial_state: new_state}\n",
    "            batch_loss, new_state, _ = sess.run([model.loss, \n",
    "                                                 model.final_state, \n",
    "                                                 model.optimizer], \n",
    "                                                 feed_dict=feed)\n",
    "            \n",
    "            end = time.time()\n",
    "            print('Epoch: {}/{}... '.format(e+1, epochs),\n",
    "                  'Training Step: {}... '.format(counter),\n",
    "                  'Training loss: {:.4f}... '.format(batch_loss),\n",
    "                  '{:.4f} sec/batch'.format((end-start)))\n",
    "        \n",
    "            if (counter % save_every_n == 0):\n",
    "                saver.save(sess, \"checkpoints/i{}_l{}.ckpt\".format(counter, lstm_size))\n",
    "    \n",
    "    saver.save(sess, \"checkpoints/i{}_l{}.ckpt\".format(counter, lstm_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Saved checkpoints\n",
    "\n",
    "Read up on saving and loading checkpoints here: https://www.tensorflow.org/programmers_guide/variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "model_checkpoint_path: \"checkpoints/i4140_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i1800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i2800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3200_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3400_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3600_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i3800_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i4000_l512.ckpt\"\n",
       "all_model_checkpoint_paths: \"checkpoints/i4140_l512.ckpt\""
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.get_checkpoint_state('checkpoints')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Sampling\n",
    "\n",
    "Now that the network is trained, we'll can use it to generate new text. The idea is that we pass in a character, then the network will predict the next character. We can use the new one, to predict the next one. And we keep doing this to generate all new text. I also included some functionality to prime the network with some text by passing in a string and building up a state from that.\n",
    "\n",
    "The network gives us predictions for each character. To reduce noise and make things a little less random, I'm going to only choose a new character from the top N most likely characters.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def pick_top_n(preds, vocab_size, top_n=5):\n",
    "    p = np.squeeze(preds)\n",
    "    p[np.argsort(p)[:-top_n]] = 0\n",
    "    p = p / np.sum(p)\n",
    "    c = np.random.choice(vocab_size, 1, p=p)[0]\n",
    "    return c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "def sample(checkpoint, n_samples, lstm_size, vocab_size, prime=\"The \"):\n",
    "    samples = [c for c in prime]\n",
    "    model = CharRNN(len(vocab), lstm_size=lstm_size, sampling=True)\n",
    "    saver = tf.train.Saver()\n",
    "    with tf.Session() as sess:\n",
    "        saver.restore(sess, checkpoint)\n",
    "        new_state = sess.run(model.initial_state)\n",
    "        for c in prime:\n",
    "            x = np.zeros((1, 1))\n",
    "            x[0,0] = vocab_to_int[c]\n",
    "            feed = {model.inputs: x,\n",
    "                    model.keep_prob: 1.,\n",
    "                    model.initial_state: new_state}\n",
    "            preds, new_state = sess.run([model.prediction, model.final_state], \n",
    "                                         feed_dict=feed)\n",
    "\n",
    "        c = pick_top_n(preds, len(vocab))\n",
    "        samples.append(int_to_vocab[c])\n",
    "\n",
    "        for i in range(n_samples):\n",
    "            x[0,0] = c\n",
    "            feed = {model.inputs: x,\n",
    "                    model.keep_prob: 1.,\n",
    "                    model.initial_state: new_state}\n",
    "            preds, new_state = sess.run([model.prediction, model.final_state], \n",
    "                                         feed_dict=feed)\n",
    "\n",
    "            c = pick_top_n(preds, len(vocab))\n",
    "            samples.append(int_to_vocab[c])\n",
    "        \n",
    "    return ''.join(samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Here, pass in the path to a checkpoint and sample from the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'checkpoints/i4140_l512.ckpt'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.latest_checkpoint('checkpoints')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Erase una vez presencia que, al cual, con todo su\n",
      "esposo, y la padre de su cana, dijo a Sancho, que los que las parece suelen\n",
      "sigo escuderos de su carto, que, por más de su mesmo desaguisado, que el\n",
      "cual, con turbada y engerida, y llevaron a prueba en la mano, y luego la\n",
      "dejó de adornos, dejando con todo cuanto en este tengo, se lo dejaron\n",
      "alegres y su aldeana; y la cual, a quien er que de su poderosa empresó se\n",
      "despidió, por la menos, con tanta male y con el castillo, solase de\n",
      "mescado, cerrada y el camino de la causa, puesto de armado conocido a los\n",
      "más respondes de la soldada, de cómo al desencantador dice:\n",
      "\n",
      "-Si es de las cueles se ha dicho, que los mesmos que su pedaz, deje mi\n",
      "señor en los día y a mi marido este donarre que te llega el diablo, y\n",
      "sabes, y nunca se hay pramio, que le de más dormir a tantas pruebas donde se\n",
      "poco a su partida, sino a más consejo, que en eso si muerto en la cabezna\n",
      "   quiera saber que más me ha hacedo\n",
      "   que mi señor de la manera quedrás\n",
      "escudero, y que si no me acuerdo de mayores,o a lo que hace más, que\n",
      "es el mesmo pelo de la ventura; pero no tuviesen los dos, que\n",
      "tenerlos para estas reinas, cuando está alegre desta vida; que si no le\n",
      "han perdido mil suspiros, con los criados que le dejan a los dos cuantos le\n",
      "parecen, para qué me han de poca el cuento y por el mismo de mi amo y alguna,\n",
      "con ellos lo que se me dice por lo que se los tiene de alcanzar la merece,\n",
      "porque ser los cortes que se están a lingo de aquéllos y parecerlas en el\n",
      "moro en la venta, para que tan bien son mucho de su señor, y, como el\n",
      "son que me han de hacer algo entre el preguntamiento: de lo que me da mi\n",
      "primer contra el sol de ti marcel como el pregunto y cura y de lo que\n",
      "sus cosas quieren, que yo le de carta el pobre podrido hablan al mesmo prado;\n",
      "que yo hay por menos, como la historia que si mi malo de escosa, que le\n",
      "ha de pudo, sino el que todos los que son los compatas de condenados; que\n",
      "lo que déste mante mi el punto que se les parezcan en ella sin el deseo de\n",
      "los caballos\n"
     ]
    }
   ],
   "source": [
    "#checkpoint = tf.train.latest_checkpoint('checkpoints')\n",
    "checkpoint = './i4140_l512.ckpt'\n",
    "samp = sample(checkpoint, 2000, lstm_size, len(vocab), prime=\"Erase una vez \")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Erase una vez ara e la peres, al a las eros asten era as ar es ciro qes\n",
      "ala que pos direr di dus las es er a lara es e enca den do le dinas de en er uon en a en an ora qe elel rue a ale que pel ore an ar lasa es liro, que e curo que e mar es dusta,, que lon os en as lerar e con oro do\n",
      "to to de les a don al lirera el la lo lo que enande, ane pore lantos or ol era al lera an ua al eno des\n",
      "ensa den as eresa qele ane poste, peno der iastos e dun ar o lu en as an lo cia antana a de core las de den ol do ceno e lo postana el due den o e a eral ane que dintias os estia do dosss linos o la as lu due do col lo quu ertera, a lo an lus an en a la an a a as liana la des le an e alta certa a duen an altara diro la co as an ar o li or os ans en a le den a alos al o lar ios a la antros qee listan en eno ane er a en o lente lo pertera an a cirrastda, yre lo lontio an a la do el den de cor en li es an es er a contentias ero de las ane, a en la due dor larto di die do qee era ar ane an lela que pira de larardo que est\n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i200_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Erase una vez \")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Erase una vez encentar suerte y\n",
      "escadado a su acasión. Y su suy a mis razonade y contiriosa de la cosa, señora don Quijote; de lo que está en todo\n",
      "la de manas y menos que se lendo del de\n",
      "lo que hasta en el minotro crayo de la mano de su pociento y en manos a\n",
      "la vilerte, y allí se acortó, y, al de señora el mano don\n",
      "Quijote, y allí con la puede de molera a la paltura y a las anchas padas, porque, si aquela para mucho\n",
      "dejarte espero en todo el menorme, se hombre de con todo los pundo de cabliller,\n",
      "camina al minos de caballero, allín, y, a qué dijo a lugeros que en las andanas y\n",
      "antreseguintimas; pero al cartallos, y a la mil de la porta contendad que le panera que no hoyo don Quijote que emos, y\n",
      "espadar lo que diese en el mano, y así\n",
      "en malando de su partencia de lo que a lo mendo a\n",
      "la\n",
      "dura de la cual su dieren, que si mis de la carrando; y, alguna de las provechas por mi mucha conazón a su pacio.\n",
      "\n",
      "-¡Es más muy bien ados, porque le amigo a su suñado de las amoses que el cuerpo.\n",
      "\n",
      "En la panta, que es los\n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i1200_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Erase una vez \")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Erase una vez suerte, y\n",
      "así le dijeron que salía cristianos y acomodadas a cuantas la verdad\n",
      "suele ella en las corazos, y dijirlo que a las cosas que se acabana de su\n",
      "caballo y dos costrades castillos, con las calles; y que aquel cosas que\n",
      "he puesto esta andante caballería, que si no hace en todos en el camino\n",
      "palabra, para que le preguntarán diciendo que la doncella en su señor deseos\n",
      "de lociras, que, sin hallarse, que siendo la pena de mi padre y el siempre de\n",
      "lo que lo dijesen que su señor había de decir, y daba con la mano, diciendo\n",
      "que aquí al dicho que sucedió don Quijote de la concertada duquesa, porque\n",
      "no le parece que estaba para que el precio de mi amo la contó a su armita que\n",
      "su caballo se las dijo:\n",
      "\n",
      "-Seguían lo que se descubrieren a la cabeza, que en la carta que luego allí\n",
      "leer a mi puerta a ser amante con ellas.\n",
      "\n",
      "»Era comedida, y estos conocidos y el del día, y el parro del sueño de\n",
      "los caballeros, y el pastor, don Gelesillo; y, así como a su puerto en ellas\n",
      "a sus caballesos, porque tod\n"
     ]
    }
   ],
   "source": [
    "checkpoint = 'checkpoints/i3400_l512.ckpt'\n",
    "samp = sample(checkpoint, 1000, lstm_size, len(vocab), prime=\"Erase una vez \")\n",
    "print(samp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}