unittest.mock -- The Wonder and the Horror.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mock: A tool for unit testing\n",
    "\n",
    "We all know the joys of unit testing. But how do you isolate the functionality you want to test when your function calls external resources, or when it uses computation-heavy steps such as machine-learning models?\n",
    "\n",
    "Mocking code paths you don't want to test can be a good way to do this. The standard library's unittest package contains a sub-package called \"mock\" which supercharges (test-related) mockery. Its central object, the `Mock`, lets you avoid writing your own stubs, and can help you verify what is and isn't being called on the mock.\n",
    "\n",
    "https://docs.python.org/3/library/unittest.mock.html\n",
    "\n",
    "ChiPy monthly meeting, 10 March 2016<br>\n",
    "Stephen Hoover<br>\n",
    "@StephenActual<br>\n",
    "https://github.com/stephen-hoover"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from unittest import mock  # Standard library\n",
    "\n",
    "# Use these for examples\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn import base"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is the Mock?\n",
    "It is whatever you want it to be."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mymock = mock.Mock()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ask it what attributes it has -- it will happily agree that everything exists, creating new Mock objects to return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Mock name='mock.an_attribute' id='4550270928'>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.an_attribute"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Mock name='mock.a_method()' id='4554675312'>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.a_method()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Mock name='mock.long.string.of.stuff.leading.to.snakes()' id='4554678056'>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.long.string.of.stuff.leading.to.snakes()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you ask for the same thing twice, you get back the same Mock object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Mock name='mock.an_attribute' id='4550270928'>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.an_attribute"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can store values inside the Mock."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam\n"
     ]
    }
   ],
   "source": [
    "mymock.create_something().value = 'spam'\n",
    "result = mymock.create_something()\n",
    "print(result.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can tell it what to return when you call things as functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
      "          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,\n",
      "          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,\n",
      "          verbose=0, warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "mymock.create_estimator.return_value = LogisticRegression()\n",
    "model = mymock.create_estimator()\n",
    "print(model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want a more opinionated mock, you can make it look like another object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mock_lr = mock.Mock(spec=LogisticRegression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "isinstance(mock_lr, LogisticRegression)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Mock name='mock.fit' id='4554741520'>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mock_lr.fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "Mock object has no attribute 'not_a_method'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-12-daad329c78f1>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmock_lr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnot_a_method\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/shoover/local/python35/lib/python3.5/unittest/mock.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    576\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_mock_methods\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    577\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_mock_methods\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mname\u001b[0m \u001b[0;32min\u001b[0m \u001b[0m_all_magics\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 578\u001b[0;31m                 \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Mock object has no attribute %r\"\u001b[0m \u001b[0;34m%\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    579\u001b[0m         \u001b[0;32melif\u001b[0m \u001b[0m_is_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    580\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: Mock object has no attribute 'not_a_method'"
     ]
    }
   ],
   "source": [
    "mock_lr.not_a_method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the \"side_effect\" attribute to make the Mock return different values for different calls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "mymock.snake.side_effect = [0, 1, ValueError('No more snakes!')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.snake()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mymock.snake()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "No more snakes!",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-16-964ff07fa892>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmymock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msnake\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/shoover/local/python35/lib/python3.5/unittest/mock.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(_mock_self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    915\u001b[0m         \u001b[0;31m# in the signature\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    916\u001b[0m         \u001b[0m_mock_self\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_mock_check_sig\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 917\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_mock_self\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_mock_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    918\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    919\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/shoover/local/python35/lib/python3.5/unittest/mock.py\u001b[0m in \u001b[0;36m_mock_call\u001b[0;34m(_mock_self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    976\u001b[0m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnext\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0meffect\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    977\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0m_is_exception\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 978\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    979\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0mDEFAULT\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    980\u001b[0m                     \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturn_value\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: No more snakes!"
     ]
    }
   ],
   "source": [
    "mymock.snake()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Mock in tests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the Mock to avoid running unnecessary external code in tests. Do we really need to verify that scikit-learn's logistic regresion works in our unit tests?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_most_probable(estimator: base.ClassifierMixin, \n",
    "                       X: np.ndarray) -> int:\n",
    "    yhat_proba = estimator.predict_proba(X)\n",
    "    i_max = np.argmax(yhat_proba[:,1])\n",
    "    return i_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def test_most_probable():\n",
    "    mock_lr = mock.Mock(spec=LogisticRegression)\n",
    "    yhat_proba = np.array([[0.1, 0.9], [0.5, 0.5], [0.01, 0.99]])\n",
    "    mock_lr.predict_proba.return_value = yhat_proba\n",
    "    test_input = np.arange(3)\n",
    "    i_max = find_most_probable(mock_lr, X=test_input)\n",
    "    \n",
    "    # Did we get the right answer?\n",
    "    assert i_max==2\n",
    "    \n",
    "    # We should be calling `predict_proba` on the input...\n",
    "    mock_lr.predict_proba.assert_called_once_with(test_input)\n",
    "    \n",
    "    # Not `predict`\n",
    "    mock_lr.predict.assert_not_called\n",
    "    \n",
    "    return True\n",
    "test_most_probable()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Patching\n",
    "Insert your Mock into other classes or modules!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def train_and_find_most_probable(estimator: base.ClassifierMixin,\n",
    "                                 X: np.ndarray,\n",
    "                                 y: np.ndarray) -> int:\n",
    "    from sklearn import cross_validation\n",
    "    yhat_proba = cross_validation.cross_val_predict(estimator, X, y)\n",
    "    i_max = np.argmax(yhat_proba[:,1])\n",
    "    return i_max\n",
    "\n",
    "@mock.patch('sklearn.cross_validation')\n",
    "def test_train_and_find_most_probable(mock_validation):\n",
    "    yhat_proba = np.array([[0.1, 0.9], [0.5, 0.5], [0.01, 0.99]])\n",
    "    mock_validation.cross_val_predict.return_value = yhat_proba\n",
    "    test_X, test_y = np.arange(3), np.zeros(3)\n",
    "    i_max = train_and_find_most_probable(None, test_X, test_y)\n",
    "    \n",
    "    # Did we get the right answer?\n",
    "    assert i_max==2\n",
    "    \n",
    "    assert mock_validation.cross_val_predict.call_count == 1\n",
    "    \n",
    "    return True\n",
    "\n",
    "test_train_and_find_most_probable()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Is there such a thing as too much mockery?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import boto3\n",
    "def upload_to_s3(fname: str, \n",
    "                 bucket: str, \n",
    "                 keypath: str,\n",
    "                 credentials: dict=None):\n",
    "    s3 = boto3.resource('s3', **(credentials or {}))\n",
    "    \n",
    "    if list(s3.Bucket(bucket).objects.filter(Prefix=keypath)):\n",
    "        raise ValueError('Target location is not empty!')\n",
    "    s3.meta.client.upload_file(fname, bucket, keypath)\n",
    "    \n",
    "@mock.patch('__main__.boto3')\n",
    "def test_upload(mock_boto):\n",
    "    upload_to_s3('local', 'mybucket', 'special/key', \n",
    "                 {'aws_secret_access_key': 'shhh'})\n",
    "\n",
    "    assert len(mock_boto.mock_calls) == 6\n",
    "\n",
    "    mock_boto.resource.assert_called_once_with(\n",
    "        's3', aws_secret_access_key='shhh')\n",
    "\n",
    "    s3 = mock_boto.resource()\n",
    "    s3.Bucket.assert_called_once_with('mybucket')\n",
    "    s3.Bucket().objects.filter.assert_called_once_with(\n",
    "        Prefix='special/key')\n",
    "    s3.meta.client.upload_file.assert_called_once_with(\n",
    "        'local', 'mybucket', 'special/key')\n",
    "    \n",
    "    return True\n",
    "    \n",
    "test_upload()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What are you really testing at this point?"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}