tutorials/plugins/tutorial_06_regression_plugins.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "molecular-moscow",
   "metadata": {},
   "source": [
    "# Regression Plugins"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "corrected-basis",
   "metadata": {},
   "source": [
    "\n",
    "**AutoPrognosis** provides a set of default predicton plugins and can be extended with any number of other plugins.\n",
    "\n",
    "In this tutorial, we will focus on the __regression__ tasks."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "multiple-annotation",
   "metadata": {},
   "source": [
    "### Plugins 101\n",
    "\n",
    "Every **AutoPrognosis plugin** must implement the **`Plugin`** interface provided by `autoprognosis/plugins/core/base_plugin.py`.\n",
    "\n",
    "Each **AutoPrognosis prediction plugin** must implement the **`PredictionPlugin`** interface provided by `autoprognosis/plugins/prediction/base.py`\n",
    "\n",
    "__Warning__ : If a plugin doesn't override all the abstract methods, it won't be loaded by the library.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "__API__ : Every prediction plugin must implement the following methods:\n",
    "- `name()` - a static method that returns the name of the plugin. e.g., neural_nets, perceptron, etc.\n",
    "\n",
    "- `subtype()` - a static method that returns the plugin's subtype. e.g., \"classification\", \"survival_analysis\" etc. It will be used for filtering the plugin in the optimization process.\n",
    "    \n",
    "- `hyperparameter_space()` - a static method that returns the hyperparameters that can be tuned during the optimization. The method will return a list of `skopt.space.Dimension` derived objects."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "auburn-hygiene",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "wanted-point",
   "metadata": {},
   "outputs": [],
   "source": [
    "# stdlib\n",
    "import sys\n",
    "import time\n",
    "import warnings\n",
    "\n",
    "# third party\n",
    "from IPython.display import HTML, display\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import metrics\n",
    "from sklearn.datasets import load_diabetes\n",
    "from sklearn.model_selection import train_test_split\n",
    "import tabulate\n",
    "from tqdm import tqdm\n",
    "\n",
    "if not sys.warnoptions:\n",
    "    warnings.simplefilter(\"ignore\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "devoted-console",
   "metadata": {},
   "source": [
    "### Loading the Regression plugins\n",
    "\n",
    "Make sure that you have installed AutoPrognosis in your workspace.\n",
    "\n",
    "You can do that by running `pip install .` in the root of the repository."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "coated-innocent",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autoprognosis absolute\n",
    "from autoprognosis.plugins.prediction.regression import Regression, RegressionPlugin\n",
    "\n",
    "regression = Regression()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "refined-booth",
   "metadata": {},
   "source": [
    "### List the existing plugins"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "incident-familiar",
   "metadata": {},
   "outputs": [],
   "source": [
    "regression.list_available()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dominican-tulsa",
   "metadata": {},
   "source": [
    "## Benchmarks\n",
    "\n",
    "For this example, we will use the [Airfoil Self-Noise Data Set](https://archive.ics.uci.edu/ml/datasets/airfoil+self-noise)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "advance-expert",
   "metadata": {},
   "source": [
    "### Loading the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "preceding-vermont",
   "metadata": {},
   "outputs": [],
   "source": [
    "# third party\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv(\n",
    "    \"https://archive.ics.uci.edu/ml/machine-learning-databases/00291/airfoil_self_noise.dat\",\n",
    "    header=None,\n",
    "    sep=\"\\\\t\",\n",
    ")\n",
    "\n",
    "\n",
    "last_col = df.columns[-1]\n",
    "\n",
    "y = df[last_col]\n",
    "X = df.drop(columns=[last_col])\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "opened-finding",
   "metadata": {},
   "source": [
    "### Duration benchmarks\n",
    "\n",
    "__About__ : This step measures the fit_predict duration for each plugin on the dataset. The times are reported in milliseconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "recovered-tonight",
   "metadata": {},
   "outputs": [],
   "source": [
    "duration = []\n",
    "\n",
    "plugins = regression.list_available()\n",
    "\n",
    "for plugin in tqdm(plugins):\n",
    "    plugin_duration = [plugin]\n",
    "    ctx = regression.get(plugin)\n",
    "\n",
    "    start = time.time() * 1000\n",
    "    ctx.fit_predict(X, y)\n",
    "\n",
    "    plugin_duration.append(round(time.time() * 1000 - start, 4))\n",
    "\n",
    "    duration.append(plugin_duration)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "numeric-million",
   "metadata": {},
   "source": [
    "### Duration(ms) results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "viral-suggestion",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(\n",
    "    HTML(\n",
    "        tabulate.tabulate(duration, headers=[\"Plugin\", \"Duration(ms)\"], tablefmt=\"html\")\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de21a377",
   "metadata": {},
   "source": [
    "### Prediction performance\n",
    "\n",
    "__Steps__\n",
    " - We train each prediction plugin on the dataset.\n",
    " - We report the R2 using cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c1ecd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# autoprognosis absolute\n",
    "from autoprognosis.utils.tester import evaluate_regression\n",
    "\n",
    "\n",
    "def get_metrics(plugin):\n",
    "    score = evaluate_regression(plugin, X, y)[\"str\"]\n",
    "    return score[\"r2\"]\n",
    "\n",
    "\n",
    "plugins = regression.list_available()\n",
    "\n",
    "metrics_headers = [\"Plugin\", \"R2\"]\n",
    "test_score = []\n",
    "\n",
    "\n",
    "for plugin in plugins:\n",
    "    fproc = regression.get(plugin)\n",
    "\n",
    "    score = get_metrics(fproc)\n",
    "\n",
    "    test_score.append([plugin, score])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d43c9fae",
   "metadata": {},
   "outputs": [],
   "source": [
    "display(HTML(tabulate.tabulate(test_score, headers=metrics_headers, tablefmt=\"html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "neural-dutch",
   "metadata": {},
   "source": [
    "# Congratulations!\n",
    "\n",
    "Congratulations on completing this notebook tutorial! If you enjoyed this and would like to join the movement towards Machine learning and AI for medicine, you can do so in the following ways!\n",
    "\n",
    "### Star AutoPrognosis on GitHub\n",
    "\n",
    "The easiest way to help our community is just by starring the Repos! This helps raise awareness of the tools we're building.\n",
    "\n",
    "- [Star AutoPrognosis](https://github.com/vanderschaarlab/autoprognosis)\n",
    "- [Star HyperImpute](https://github.com/vanderschaarlab/hyperimpute)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}