svm.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# SVM Kernel Approximation\n",
    "\n",
    "1. Environment Setup: import required libraries\n",
    "2. Loading Dataset and Preprocessing Data\n",
    "3. Build Input Fn\n",
    "4. Training Model\n",
    "5. Evaluating Training Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Environment Setup: import required library\n",
    "\n",
    "We include the required libraries that will be used in the next parts. The **time**, **numpy**, and **tensorflow** are common libraries in machine learning. The **fio**, which is \"file input output\" used to load data and **config**, which is \"configuration file\" used to config the path of the dataset files are written by myself. Modify it when you need it.\n",
    "\n",
    "**patch/metrics.py**: fix `tf.metrics.true_negatives` method is missing on Tensorflow r1.4."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:469: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:470: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:471: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:472: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:473: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:476: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/importlib/_bootstrap.py:219: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "import fio\n",
    "import preprocessing as pc\n",
    "from config import *\n",
    "import utility\n",
    "\n",
    "# python std library\n",
    "import gc\n",
    "import time\n",
    "import logging\n",
    "import functools\n",
    "import collections\n",
    "from multiprocessing.pool import ThreadPool\n",
    "\n",
    "# install library\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "\n",
    "# patch\n",
    "import patch.metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Loading Dataset and Preprocessing Data\n",
    "\n",
    "Loading the training set, validation set, and test set that was defined in **config.py** file. Sample files consisting of indices of data will be used to undersample the data.\n",
    "\n",
    "There are two parts of the data that must be processed:\n",
    "\n",
    "- the input data represented by the prefix \"**X**\" on variables\n",
    "- the target data represented by the prefix \"**Y**\" on variables\n",
    "\n",
    "Check out the file **preprocess.py** for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load(window_size):\n",
    "    X_train = fio.load_file(X_train_dataset)\n",
    "    Y_train = fio.load_file(Y_train_dataset)\n",
    "    X_test = fio.load_file(X_test_dataset)\n",
    "    Y_test = fio.load_file(Y_test_dataset)\n",
    "    train_sample = fio.load_sample_file(train_sample_dataset)\n",
    "    valid_sample = fio.load_sample_file(valid_sample_dataset)\n",
    "\n",
    "    stat = pc.get_feat_stat(X_train)\n",
    "    \n",
    "    X_train = pc.standardize(X_train, stat)\n",
    "    X_test = pc.standardize(X_test, stat)\n",
    "    \n",
    "    X_train = pc.expand(X_train, window_size)\n",
    "    X_test = pc.expand(X_test, window_size)\n",
    "    \n",
    "    Y_train = pc.classify(Y_train)\n",
    "    Y_test = pc.classify(Y_test)\n",
    "    \n",
    "    testing_sample = [np.indices((x.shape[0], x.shape[1])).reshape((2,-1)).T for x in X_test]\n",
    "    \n",
    "    return X_train, Y_train, X_test, Y_test, train_sample, valid_sample, testing_sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the length of training set: 3\n",
      "the length of testing set: 2\n",
      "the first row training data: 0.60705996\n",
      "the first row target value: 1\n",
      "(5, 3, 23, 23, 6)\n",
      "(5, 3, 1)\n",
      "[[0 0]\n",
      " [3 0]\n",
      " [4 0]\n",
      " [0 1]\n",
      " [1 1]\n",
      " [2 1]\n",
      " [3 1]\n",
      " [4 1]\n",
      " [0 2]\n",
      " [2 2]\n",
      " [3 2]\n",
      " [4 2]]\n",
      "[[1 0]\n",
      " [2 0]\n",
      " [1 2]]\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    window_size = 23\n",
    "    \n",
    "    X_train_orig, Y_train_orig, X_test_orig, Y_test_orig, train_sample, valid_sample, testing_sample = load(window_size)\n",
    "    print(\"the length of training set:\", len(X_train_orig))\n",
    "    print(\"the length of testing set:\", len(X_test_orig))\n",
    "    print(\"the first row training data:\", np.sum(X_train_orig[0][0][0]))\n",
    "    print(\"the first row target value:\", np.sum(Y_train_orig[0][0][0]))\n",
    "    \n",
    "    print(X_train_orig[0].shape)\n",
    "    print(Y_train_orig[0].shape)\n",
    "    print(train_sample[0])\n",
    "    print(valid_sample[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Build Input Fn\n",
    "\n",
    "- [Tensorflow Doc: dataset.from_generator](https://github.com/tensorflow/docs/blob/r1.4/site/en/api_docs/api_docs/python/tf/data/Dataset.md#from_generator)\n",
    "- [Tensorflow Doc: dataset.batch](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#batch)\n",
    "    - batch_size: A tf.int64 scalar tf.Tensor, representing the number of consecutive elements of this dataset to combine in a single batch.\n",
    "    - drop_remainder: (Optional.) A tf.bool scalar tf.Tensor, representing whether the last batch should be dropped in the case it has fewer than batch_size elements; the default behavior is not to drop the smaller batch.\n",
    "- [Tensorflow Doc: dataset.padded_batch](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#padded_batch)\n",
    "- [How to use dataset in tensorflow](https://towardsdatascience.com/how-to-use-dataset-in-tensorflow-c758ef9e4428)\n",
    "- [StackOverflow: Train Tensorflow model with estimator (from_generator)](https://stackoverflow.com/questions/49673602/train-tensorflow-model-with-estimator-from-generator?rq=1)\n",
    "- [StackOverflow: Is Tensorflow Dataset API slower than Queues?](https://stackoverflow.com/questions/47403407/is-tensorflow-dataset-api-slower-than-queues)\n",
    "- [Github: How can I ues Dataset to shuffle a large whole dataset?](https://github.com/tensorflow/tensorflow/issues/14857)\n",
    "\n",
    "**Got the warning: Out of range StopIteration**\n",
    "\n",
    "```shell\n",
    "W tensorflow/core/framework/op_kernel.cc:1192] Out of range: StopIteration: Iteration finished\n",
    "```\n",
    "\n",
    "> I also meeting this problem same for you,but it is not a bug.\n",
    ">\n",
    "> you can see the doc in https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator about train()\n",
    "> \n",
    "> steps: Number of steps for which to train model. If None, train forever or train until input_fn generates the OutOfRange error or StopIteration exception. 'steps' works incrementally. If you call two times train(steps=10) then training occurs in total 20 steps. If OutOfRange or StopIteration occurs in the middle, training stops before 20 steps. If you don't want to have incremental behavior please set max_steps instead. If set, max_steps must be None.\n",
    ">\n",
    "> -- libulin\n",
    "\n",
    "From [Github Comment](https://github.com/tensorflow/tensorflow/issues/12414#issuecomment-345131765)\n",
    "\n",
    "With the fix in [301a6c4](https://github.com/tensorflow/tensorflow/commit/301a6c41cbb111fae89657a49775920aa70525fd) (and a related fix for the StopIteration logging in [c154d47](https://github.com/tensorflow/tensorflow/commit/c154d4719eea88e694f4c06bcb1249dbac0f7877), the logs should be much quieter when using tf.data.\n",
    "\n",
    "Simple fix:\n",
    "\n",
    "```python\n",
    "import os\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'  # ERROR\n",
    "import tensorflow as tf\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def np_input_fn(X, Y, samples, shuffle=False, window_size=1, batch=None, epoch=None):\n",
    "    if batch is None:\n",
    "        raise Exception('batch can not be None')\n",
    "    \n",
    "    if window_size & 0x1 == 0:\n",
    "        raise Exception('window size can not even')\n",
    "    dim_in = window_size * window_size * 6\n",
    "    \n",
    "    if not isinstance(X, list):\n",
    "        X = [X]\n",
    "    if not isinstance(Y, list):\n",
    "        Y = [Y]\n",
    "    if not isinstance(samples, list):\n",
    "        samples = [samples]\n",
    "    \n",
    "    samples = [np.pad(x, ((0,0),(1,0)), 'constant', constant_values=i) for i, x in enumerate(samples)]\n",
    "    samples = np.concatenate(samples)\n",
    "    \n",
    "    print(\"input_fn total size\", len(samples))\n",
    "    \n",
    "    def generator():\n",
    "        if shuffle == True:\n",
    "            np.random.shuffle(samples)\n",
    "        \n",
    "        for s in samples:\n",
    "            x = X[s[0]][s[1], s[2]].reshape((dim_in))\n",
    "            y = Y[s[0]][s[1], s[2]].reshape((1))\n",
    "            yield x, y\n",
    "    \n",
    "    def _input_fn():\n",
    "        dataset = tf.data.Dataset.from_generator(generator,\n",
    "                                                   output_types= (tf.float32, tf.int32), \n",
    "                                                   output_shapes=(tf.TensorShape([dim_in]), tf.TensorShape([1])))\n",
    "        dataset = dataset.batch(batch_size=batch)\n",
    "        dataset = dataset.repeat(epoch)\n",
    "        dataset = dataset.prefetch(1)\n",
    "\n",
    "        iterator = dataset.make_one_shot_iterator()\n",
    "        features_tensors, labels = iterator.get_next()\n",
    "        print(features_tensors)\n",
    "        print(labels)\n",
    "        features = {'data': features_tensors }\n",
    "        return features, labels\n",
    "    \n",
    "    return _input_fn\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_fn total size 16092\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "({'data': <tf.Tensor 'IteratorGetNext:0' shape=(?, 3174) dtype=float32>}, <tf.Tensor 'IteratorGetNext:1' shape=(?, 1) dtype=int32>)\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    print(np_input_fn(X_train_orig, Y_train_orig, train_sample, window_size=23, batch=128)())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Training Model\n",
    "\n",
    "The model that we used is followed by the article: [Improving Linear Models Using Explicit Kernel Methods](https://github.com/Debian/tensorflow/blob/master/tensorflow/contrib/kernel_methods/g3doc/tutorial.md).\n",
    "\n",
    "[TensorFlow Estimators: Managing Simplicity vs. Flexibility in\n",
    "High-Level Machine Learning Frameworks](https://storage.googleapis.com/pub-tools-public-publication-data/pdf/18d86099a350df93f2bd88587c0ec6d118cc98e7.pdf)\n",
    "\n",
    "Optimizer\n",
    "\n",
    "- [Ftrl Optimizer](https://www.tensorflow.org/api_docs/python/tf/compat/v1/train/FtrlOptimizer)\n",
    "- [Adam Optimizer](https://www.tensorflow.org/api_docs/python/tf/compat/v1/train/AdamOptimizer)\n",
    "\n",
    "Build the following models:\n",
    "\n",
    "1. Build Linear Classifier Model\n",
    "2. Build Random Fourier Feature Mapper Model and Linear Classifier Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Training Linear Classifier Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_linear_model(learning_rate, dim_in, config=None):\n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "    image_column = tf.contrib.layers.real_valued_column('data', dimension=dim_in)\n",
    "    \n",
    "    estimator = tf.contrib.learn.LinearClassifier(\n",
    "        feature_columns=[image_column],\n",
    "        n_classes=2, \n",
    "        config=config,\n",
    "        optimizer=optimizer)\n",
    "\n",
    "    return estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_fn total size 16092\n",
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmpks701peg\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x827ed7390>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': '/var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmpks701peg'}\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "WARNING:tensorflow:From /Users/akiicat/opt/anaconda3/envs/universe/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/estimators/linear.py:173: get_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please switch to tf.train.get_global_step\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmpks701peg/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.6931475, step = 1\n",
      "INFO:tensorflow:global_step/sec: 39.7969\n",
      "INFO:tensorflow:loss = 0.76422954, step = 101 (2.517 sec)\n",
      "INFO:tensorflow:global_step/sec: 43.0958\n",
      "INFO:tensorflow:loss = 0.6497522, step = 201 (2.318 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 252 into /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmpks701peg/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.6976804.\n",
      "Elapsed time: 7.861431121826172 seconds\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    batch = 128\n",
    "    epoch = 2\n",
    "    train_input_fn = np_input_fn(X_train_orig, Y_train_orig, train_sample, shuffle=True, window_size=23, batch=batch, epoch=epoch)\n",
    "    \n",
    "    learning_rate = 0.001       # Adam Optimizer\n",
    "    input_dim = 23 * 23 * 6     # Data size\n",
    "    \n",
    "    estimator = create_linear_model(learning_rate, input_dim)\n",
    "\n",
    "    start = time.time()\n",
    "    estimator.fit(input_fn=train_input_fn) # Train.\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Training Random Fourier Feature Mapper Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_rffm_model(learning_rate, dim_in, dim_out, stddev, config=None):\n",
    "    \n",
    "    kernel_mapper = tf.contrib.kernel_methods.RandomFourierFeatureMapper(dim_in, dim_out, stddev, name='rffm')\n",
    "    \n",
    "    optimizer = tf.train.AdamOptimizer(learning_rate)\n",
    "    image_column = tf.contrib.layers.real_valued_column('data', dimension=dim_in)\n",
    "\n",
    "    estimator = tf.contrib.kernel_methods.KernelLinearClassifier(\n",
    "        feature_columns=[image_column], \n",
    "        n_classes=2, \n",
    "        config=config,\n",
    "        optimizer=optimizer, \n",
    "        kernel_mappers={image_column: [kernel_mapper]})\n",
    "    \n",
    "    return estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_fn total size 16092\n",
      "INFO:tensorflow:Using default config.\n",
      "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx\n",
      "INFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x828faa128>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "}\n",
      ", '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_secs': 600, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': None, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': '/var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx'}\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Create CheckpointSaverHook.\n",
      "INFO:tensorflow:Saving checkpoints for 1 into /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx/model.ckpt.\n",
      "INFO:tensorflow:loss = 0.6931475, step = 1\n",
      "INFO:tensorflow:global_step/sec: 1.32429\n",
      "INFO:tensorflow:loss = 0.6822852, step = 101 (75.511 sec)\n",
      "INFO:tensorflow:Saving checkpoints for 126 into /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx/model.ckpt.\n",
      "INFO:tensorflow:Loss for final step: 0.70227647.\n",
      "Elapsed time: 247.24206590652466 seconds\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    batch = 128\n",
    "    epoch = 1\n",
    "    train_input_fn = np_input_fn(X_train_orig, Y_train_orig, train_sample, shuffle=True, window_size=23, batch=batch, epoch=epoch)\n",
    "    \n",
    "    learning_rate = 0.001  # Adam Optimizer\n",
    "\n",
    "    # RFFM\n",
    "    input_dim = 23 * 23 * 6\n",
    "    output_dim = 23 * 23 * 6 * 10\n",
    "    stddev = 1.0\n",
    "\n",
    "    estimator = create_rffm_model(learning_rate, input_dim, output_dim, stddev)\n",
    "    \n",
    "    start = time.time()\n",
    "    estimator.fit(input_fn=train_input_fn) # Train.\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Evaluating Training Data\n",
    "\n",
    "1. Evaluating Training Data\n",
    "2. Evaluating Validation Data\n",
    "3. Evaluating Testing Data\n",
    "\n",
    "**Confusion Matrix**\n",
    "\n",
    "- [Classification: True vs. False and Positive vs. Negative](https://developers.google.com/machine-learning/crash-course/classification/true-false-positive-negative)\n",
    "- [如何辨別機器學習模型的好壞？秒懂Confusion Matrix](https://www.ycc.idv.tw/confusion-matrix.html)\n",
    "\n",
    "**estimator.evaluate**\n",
    "\n",
    "- [Tensorflow Doc: estimator evaluate metrics](https://tensorflow.google.cn/versions/r1.15/api_docs/python/tf/keras/metrics)\n",
    "- [Stack Overflow: Meaning of evaluation metrics in Tensorflow](https://ai.stackexchange.com/questions/6383/meaning-of-evaluation-metrics-in-tensorflow)\n",
    "\n",
    "```python\n",
    "x, y = {'data': X}, Y\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(x, y, batch_size=batch, shuffle=False, num_epochs=1)\n",
    "metric = estimator.evaluate(input_fn=input_fn)\n",
    "```\n",
    "\n",
    "**estimator.predict_classes**\n",
    "\n",
    "```python\n",
    "x, y = {'data': X_train.astype(np.float32) }, Y_train\n",
    "batch = 128\n",
    "\n",
    "input_fn = tf.estimator.inputs.numpy_input_fn(x, batch_size=batch, shuffle=False, num_epochs=1)\n",
    "metric = estimator.predict_classes(input_fn=input_fn)\n",
    "\n",
    "for i, p in enumerate(metric):\n",
    "    print(p, y[i][0])\n",
    "```\n",
    "\n",
    "**Metrics**\n",
    "\n",
    "- [Python tensorflow.contrib.learn.MetricSpec() Examples](https://www.programcreek.com/python/example/96156/tensorflow.contrib.learn.MetricSpec)\n",
    "- [Tensorflow Doc: Available Metrics](https://github.com/tensorflow/docs/tree/r1.4/site/en/api_docs/api_docs/python/tf/metrics)\n",
    "\n",
    "```python\n",
    "metrics = { \"accuracy\": learn.MetricSpec(metric_fn=tf.metrics.accuracy, prediction_key=\"classes\") }\n",
    "metric = estimator.evaluate(input_fn=eval_input_fn, metrics=metrics)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_model(estimator, X, Y, samples, window_size=1, batch=2048, epoch=1):\n",
    "\n",
    "    eval_input_fn = np_input_fn(X, Y, samples, shuffle=False, window_size=window_size, batch=batch, epoch=epoch)\n",
    "    \n",
    "    metrics = {\n",
    "        \"tp\": tf.contrib.learn.MetricSpec(metric_fn=tf.metrics.true_positives, prediction_key=\"classes\"),\n",
    "        \"tn\": tf.contrib.learn.MetricSpec(metric_fn=patch.metrics.true_negatives, prediction_key=\"classes\"),\n",
    "        \"fp\": tf.contrib.learn.MetricSpec(metric_fn=tf.metrics.false_positives, prediction_key=\"classes\"),\n",
    "        \"fn\": tf.contrib.learn.MetricSpec(metric_fn=tf.metrics.false_negatives, prediction_key=\"classes\"),\n",
    "    }\n",
    "    \n",
    "    start = time.time()\n",
    "    metric = estimator.evaluate(input_fn=eval_input_fn, metrics=metrics)\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))\n",
    "    \n",
    "    return metric\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 Evaluating Training Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_fn total size 16092\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2020-03-24-20:46:54\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx/model.ckpt-126\n",
      "INFO:tensorflow:Finished evaluation at 2020-03-24-20:55:26\n",
      "INFO:tensorflow:Saving dict for global step 126: accuracy = 0.68207806, accuracy/baseline_label_mean = 0.5036664, accuracy/threshold_0.500000_mean = 0.68207806, auc = 0.75058556, auc_precision_recall = 0.74787956, fn = 2484.0, fp = 2632.0, global_step = 126, labels/actual_label_mean = 0.5036664, labels/prediction_mean = 0.5025797, loss = 0.61646944, precision/positive_threshold_0.500000_mean = 0.68108565, recall/positive_threshold_0.500000_mean = 0.6935225, tn = 5355.0, tp = 5621.0\n",
      "Elapsed time: 528.6947410106659 seconds\n",
      "Elapsed time: 528.7258548736572 seconds\n",
      "training metrics: {'loss': 0.61646944, 'accuracy': 0.68207806, 'labels/prediction_mean': 0.5025797, 'labels/actual_label_mean': 0.5036664, 'accuracy/baseline_label_mean': 0.5036664, 'auc': 0.75058556, 'auc_precision_recall': 0.74787956, 'accuracy/threshold_0.500000_mean': 0.68207806, 'precision/positive_threshold_0.500000_mean': 0.68108565, 'recall/positive_threshold_0.500000_mean': 0.6935225, 'fn': 2484.0, 'fp': 2632.0, 'tn': 5355.0, 'tp': 5621.0, 'global_step': 126}\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    start = time.time()\n",
    "    metrics = evaluate_model(estimator, X_train_orig, Y_train_orig, train_sample, batch=1, window_size=23)\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))\n",
    "    print(\"training metrics:\", metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 Evaluating Validation Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input_fn total size 3973\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2020-03-24-20:55:41\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx/model.ckpt-126\n",
      "INFO:tensorflow:Finished evaluation at 2020-03-24-20:58:01\n",
      "INFO:tensorflow:Saving dict for global step 126: accuracy = 0.49811226, accuracy/baseline_label_mean = 0.49836394, accuracy/threshold_0.500000_mean = 0.49811226, auc = 0.49631155, auc_precision_recall = 0.48840523, fn = 982.0, fp = 1012.0, global_step = 126, labels/actual_label_mean = 0.49836394, labels/prediction_mean = 0.5020263, loss = 0.72332865, precision/positive_threshold_0.500000_mean = 0.49651742, recall/positive_threshold_0.500000_mean = 0.5040404, tn = 981.0, tp = 998.0\n",
      "Elapsed time: 145.8856339454651 seconds\n",
      "Elapsed time: 145.89155507087708 seconds\n",
      "validation metrics: {'loss': 0.72332865, 'accuracy': 0.49811226, 'labels/prediction_mean': 0.5020263, 'labels/actual_label_mean': 0.49836394, 'accuracy/baseline_label_mean': 0.49836394, 'auc': 0.49631155, 'auc_precision_recall': 0.48840523, 'accuracy/threshold_0.500000_mean': 0.49811226, 'precision/positive_threshold_0.500000_mean': 0.49651742, 'recall/positive_threshold_0.500000_mean': 0.5040404, 'fn': 982.0, 'fp': 1012.0, 'tn': 981.0, 'tp': 998.0, 'global_step': 126}\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    start = time.time()\n",
    "    metrics = evaluate_model(estimator, X_train_orig, Y_train_orig, valid_sample, batch=1, window_size=23)\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))\n",
    "    print(\"validation metrics:\", metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.3 Evaluating Testing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n",
      "(4, 3, 23, 23, 6)\n",
      "(20, 10, 23, 23, 6)\n",
      "input_fn total size 212\n",
      "Tensor(\"IteratorGetNext:0\", shape=(?, 3174), dtype=float32)\n",
      "Tensor(\"IteratorGetNext:1\", shape=(?, 1), dtype=int32)\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "WARNING:tensorflow:Casting <dtype: 'int32'> labels to bool.\n",
      "INFO:tensorflow:Starting evaluation at 2020-03-24-20:58:07\n",
      "INFO:tensorflow:Restoring parameters from /var/folders/40/1jhkx0kj6ld3fv5st6wpccsr0000gp/T/tmp15r9t5tx/model.ckpt-126\n",
      "INFO:tensorflow:Finished evaluation at 2020-03-24-20:58:29\n",
      "INFO:tensorflow:Saving dict for global step 126: accuracy = 0.49056605, accuracy/baseline_label_mean = 0.509434, accuracy/threshold_0.500000_mean = 0.49056605, auc = 0.5028935, auc_precision_recall = 0.5129694, fn = 51.0, fp = 57.0, global_step = 126, labels/actual_label_mean = 0.509434, labels/prediction_mean = 0.5030061, loss = 0.701244, precision/positive_threshold_0.500000_mean = 0.5, recall/positive_threshold_0.500000_mean = 0.5277778, tn = 47.0, tp = 57.0\n",
      "Elapsed time: 28.010125875473022 seconds\n",
      "Elapsed time: 28.011068105697632 seconds\n",
      "testing metrics: {'loss': 0.701244, 'accuracy': 0.49056605, 'labels/prediction_mean': 0.5030061, 'labels/actual_label_mean': 0.509434, 'accuracy/baseline_label_mean': 0.509434, 'auc': 0.5028935, 'auc_precision_recall': 0.5129694, 'accuracy/threshold_0.500000_mean': 0.49056605, 'precision/positive_threshold_0.500000_mean': 0.5, 'recall/positive_threshold_0.500000_mean': 0.5277778, 'fn': 51.0, 'fp': 57.0, 'tn': 47.0, 'tp': 57.0, 'global_step': 126}\n"
     ]
    }
   ],
   "source": [
    "if __name__ == \"__main__\":\n",
    "    print(len(X_test_orig))\n",
    "    print(X_test_orig[0].shape)\n",
    "    print(X_test_orig[1].shape)\n",
    "    start = time.time()\n",
    "    metrics = evaluate_model(estimator, X_test_orig, Y_test_orig, testing_sample, batch=1, window_size=23)\n",
    "    end = time.time()\n",
    "    print('Elapsed time: {} seconds'.format(end - start))\n",
    "    print(\"testing metrics:\", metrics)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}