notebooks/Syft - Testing - Benchmark Tests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing: Benchmark Tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One goal of the OpenMined project is to **efficiently** train Deep Learning models in a homomorphically encrypted state. Therefore it is very important to benchmark new and existing features in order to achieve better and faster implementations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installation\n",
    "\n",
    "Simpy run `pip install -r test-requirements.txt` instead of the regular `requirements.txt` to get all testing tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before using the Benchmark Testing Suite, you have to import it from `syft.test.benchmark`. After that, you can pass in the function which needs benchmark testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<syft.test.benchmark.Benchmark at 0x10e930160>"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from syft.test.benchmark import Benchmark\n",
    "\n",
    "Benchmark(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Compute a function's execution time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "The `exec_time` method is a very basic tool to calculate a functions' execution time. The method can be used as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EXECUTION TIME: [3.0043067932128906] SECONDS\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "def wait_a_second(seconds=3):  # Define a function for testing or use an existing one\n",
    "    time.sleep(seconds)\n",
    "    \n",
    "# Call function without params\n",
    "exec_time = Benchmark(wait_a_second).exec_time()\n",
    "\n",
    "print(\"EXECUTION TIME: {} SECONDS\".format(exec_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So as we see, the function returns the execution time of the function in seconds. Additional params can be added to the function call as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2) EXECUTION TIME: [1.0004808902740479] SECONDS\n"
     ]
    }
   ],
   "source": [
    "# Call functions with params\n",
    "exec_time = Benchmark(wait_a_second, seconds=1).exec_time()  # Pass function and params to be tested into class\n",
    "\n",
    "print(\"(2) EXECUTION TIME: {} SECONDS\".format(exec_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Profile Execution Times Per Line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is possible to get the execution time per line using the `profile_lines()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Timer unit: 1e-06 s\n",
      "\n",
      "Total time: 3.1e-05 s\n",
      "File: <ipython-input-4-4317e0387243>\n",
      "Function: some_function at line 1\n",
      "\n",
      "Line #      Hits         Time  Per Hit   % Time  Line Contents\n",
      "==============================================================\n",
      "     1                                           def some_function(count):\n",
      "     2         1            7      7.0     22.6      a = 6*8\n",
      "     3         1            2      2.0      6.5      b = 6**3\n",
      "     4         1            2      2.0      6.5      c = a + b\n",
      "     5         1            2      2.0      6.5      x = [a, b, c]\n",
      "     6         6            8      1.3     25.8      for i in range(count):\n",
      "     7         5           10      2.0     32.3          a += x[0] * i + x[1] + x [2] \n",
      "\n"
     ]
    }
   ],
   "source": [
    "def some_function(count):\n",
    "    a = 6*8\n",
    "    b = 6**3\n",
    "    c = a + b\n",
    "    x = [a, b, c]\n",
    "    for i in range(count):\n",
    "        a += x[0] * i + x[1] + x [2] \n",
    "\n",
    "Benchmark(some_function, count=5).profile_lines()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}