hw_formatting_guide.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Homework and bake-off code: Formatting guide"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "__author__ = \"Insop\"\n",
    "__version__ = \"CS224u, Stanford, Spring 2021\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Overview](#Overview)\n",
    "1. [Original system code](#Original-system-code)\n",
    "  1. [Modifying provided code in the original notebook](#Modifying-provided-code-in-the-original-notebook)\n",
    "  1. [External imports](#External-imports)\n",
    "  1. [Custom code](#Custom-code)\n",
    "  1. [Long running test code](#Long-running-test-code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "This notebook provides a list of Dos and Don'ts for writing code for original systems and bake-offs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Original system code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our assignments need to handle specific homework questions and also very open ended original systems that can have arbitrary dependencies and data requirements, so our instructions have to be quite detailed to handle both. \n",
    "\n",
    "Here's one quick reminder/clarification of a common issue:\n",
    "\n",
    "Please be sure to include your Original System code and bake-off call within the scope of this `if` conditional:\n",
    "\n",
    "```\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    test_evaluate_pooled_bert(evaluate_pooled_bert)\n",
    "```\n",
    "\n",
    "This ensures that the autograder **does not** attempt to run your original system code. This includes any `import` statements used in your Original System – they should be within the `if` conditional. \n",
    "\n",
    "Overall – please do not modify any portion of these cells other than  \n",
    " \n",
    "1. the comment spaces for system text description and peak score reporting; and \n",
    "2. the space in the `if` conditional where you are meant to put your code.\n",
    "\n",
    "Since we encourage creativity and do not want to constrain things, your original system code will instead be awarded credit manually by CFs after the assignment due date. This is also why you will not see a full grade out of 10 until after the submission deadline, when CFs have manually awarded the original system points."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifying provided code in the original notebook\n",
    "\n",
    "Please do not modify provided code in the original notebook, such as changing the function arguments or default parameters. The autograder will call functions to test the homework problem code, and the autograder uses the function arguments as shown in the original notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an example (from [hw_colors.ipynb](hw_colors.ipynb)) where the provided code was modified to use `func(vocab, 'data/glove.6B/glove.6B.50d.txt')` instead of the original code `func(vocab, 'glove.6B.50d.txt')`. This might work fine in your local environment; however, the autograder will separately call `func` the same way as shown in the original notebook. That's why we suggest you to not modify the provided code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_create_glove_embedding(func):\n",
    "    vocab = ['NLU', 'is', 'the', 'future', '.', '$UNK', '<s>', '</s>']\n",
    "\n",
    "    # DON'T modify functions like this!\n",
    "    #\n",
    "    # glove_embedding, glove_vocab = func(vocab, 'data/glove.6B/glove.6B.50d.txt')\n",
    "\n",
    "    # DO KEEP the code as it was, since the autograder calls functions in\n",
    "    # the same way shown in this line:\n",
    "    glove_embedding, glove_vocab = func(vocab, 'glove.6B.50d.txt')\n",
    "\n",
    "    assert isinstance(glove_embedding, np.ndarray), \\\n",
    "        \"Expected embedding type {}; got {}\".format(\n",
    "        glove_embedding.__class__.__name__, glove_embedding.__class__.__name__)\n",
    "    assert glove_embedding.shape == (8, 50), \\\n",
    "        \"Expected embedding shape (8, 50); got {}\".format(glove_embedding.shape)\n",
    "    assert glove_vocab == vocab, \\\n",
    "        \"Expected vocab {}; got {}\".format(vocab, glove_vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### External imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DON'T!\n",
    "#\n",
    "# This will cause the autograder to fail!\n",
    "\n",
    "pip install 'git+https://github.com/NVIDIA/dllogger'\n",
    "\n",
    "# Directly importing external modules outside of `if 'IS_GRADESCOPE_ENV'` scope\n",
    "# will also cause the autograder to fail."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DO!\n",
    "#\n",
    "# This is good!\n",
    "#\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    # You can install and import modules of your choice --\n",
    "    # for example:\n",
    "    # https://github.com/NVIDIA/dllogger/issues/1\n",
    "    pip install 'git+https://github.com/NVIDIA/dllogger'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DON'T!\n",
    "#\n",
    "# This type of custom code will fail, since the autograder is not\n",
    "# equipped with a GPU:\n",
    "#\n",
    "try:\n",
    "    t_gpu = torch.randn(3,3, device='cuda:0')\n",
    "except AssertionError as err:\n",
    "    print(err)\n",
    "t_gpu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DO\n",
    "#\n",
    "# This is good!\n",
    "#\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    # This is okay since this code will not run in the autograder\n",
    "    # environment:\n",
    "    try:\n",
    "        t_gpu = torch.randn(3,3, device='cuda:0')\n",
    "    except AssertionError as err:\n",
    "        print(err)\n",
    "    t_gpu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Long running test code\n",
    "\n",
    "Any long running test code should be inside the `if` conditional block."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DON'T!\n",
    "#\n",
    "# This type of custom code will cause the autograder to time out:\n",
    "#\n",
    "my_test_function_runs_an_hour()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#\n",
    "# DO\n",
    "#\n",
    "# This is good!\n",
    "#\n",
    "if 'IS_GRADESCOPE_ENV' not in os.environ:\n",
    "    # Run as many tests as you wish!\n",
    "    my_test_function_runs_an_hour()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}