FinalDemo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro\n",
    "\n",
    "This is a demo of the paper Teachable Reinforcement Learning via Advice Distillation showing how humans can use advice to coach agents through new tasks!\n",
    "\n",
    "For more details, check out our NeurIPS paper and video: https://neurips.cc/Conferences/2021/Schedule?showEvent=27834"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup - do this once\n",
    "\n",
    "To avoid version conflicts, we recommend running this in a conda env with python 3.7.\n",
    "\n",
    "    conda create --name teachable_rl python=3.7\n",
    "    conda activate teachable_rl\n",
    "    pip install notebook\n",
    "    \n",
    "You either need to run this on a device with a display. If you're running on a machine without one, use port forwarding:\n",
    "\n",
    "    ssh -L 9999:localhost:9999 INSERT_SERVER_NAME\n",
    "    jupyter notebook --no-browser --port 9999\n",
    "\n",
    "\n",
    "We use two environments: [BabyAI](https://github.com/mila-iqia/babyai) and [AntMaze](https://github.com/rail-berkeley/d4rl).  If you would like to use AntMaze, please [install Mujoco](https://github.com/openai/mujoco-py)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "!git clone https://github.com/aliengirlliv/teachable 1> /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "cd teachable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true,
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install -r reqs.txt 1> /dev/null"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup - Do this each time you reload the notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%matplotlib tk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd teachable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from final_demo import *\n",
    "from IPython.display import HTML\n",
    "import pathlib\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib as mpl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup\n",
    "\n",
    "## Instructions\n",
    "1. Select the collection mode.\n",
    "    - \"Advice\" runs the \"Improvement\" phase of our method, allowing you to coach an agent using waypoint advice\n",
    "    - \"Demos\" lets you collect trajectories providing actions each timestep.\n",
    "2. Select a save name (any string describing this experiment).\n",
    "3. Collect demos below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Collection\n",
    "To collect data, run the block below. A window will open which lets you collect data.  \n",
    "\n",
    "In our human exps, we found you can reach okay performance (> 50% success) on this env with about 30 mins of human collection time.\n",
    "\n",
    "## Task\n",
    "\n",
    "**BabyAI**:  The agent's task is to unlock a door by collecting a matching-colored key and using it on the corresponding door. (To speed up training time, we always spawn the agent in the same room as the locked door, and the key is always in the same spot.)\n",
    "\n",
    "**Ant**: The agent's task is to reach the pink target.\n",
    "\n",
    "### Using Advice\n",
    "\n",
    "Note: Collecting advice requires a mouse.\n",
    "\n",
    "**BabyAI**: The agent you will be coaching has been pretrained to understand Waypoint advice. It has never seen this particular environment/task before, and has never had to unlock a door. Click on a square to tell the agent to head there and manipulate any item present. Use the scrollwheel to advance.\n",
    "\n",
    "**Ant**: The agent you will be coaching has been pretrained to understand Waypoint advice. It has never seen an environment this large before. Click on a point to tell the agent to head there. Use the scrollwheel to advance.\n",
    "\n",
    "### Providing Demos (BabyAI env only)\n",
    "Use the arrow keys to navigate, Page Up/Down to manipulate objects, and Space to open doors.\n",
    "\n",
    "## Using Pre-collected data\n",
    "We include a buffer of data collected using 30 mins of human time using Advice. You can only load this data if you have CUDA enabled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "env_type = 'Ant'  # Options are 'BabyAI', 'Ant'\n",
    "collect_type = 'Advice'  # Options are 'Advice', 'Demos', or 'Precollected'\n",
    "save_path = 'temp3'  # Any string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "collector = HumanFeedback(env_type=env_type, collect_type=collect_type, \n",
    "                          save_path=save_path, seed=124)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Train\n",
    "\n",
    "Here, we train an advice-free policy on the collected trajectories using the buffer of collected trajectories.\n",
    "\n",
    "\n",
    "It will train for 20 itrs, but feel free to pause it before then if you'd like to see thet trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "args = make_args(collector, save_path)  \n",
    "run_experiment(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visualize\n",
    "\n",
    "Play a video of the agent you trained. This agent was trained using the coached rollouts you provided.  This agent does **not** receive advice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "display_trained_model(save_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the agent's success rate during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot(save_path)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}