02-experiment-tracking/week02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "356ee97d",
   "metadata": {},
   "source": [
    "# Week 2\n",
    "    \n",
    "## Important Concepts\n",
    "* ML experiment: The process of building an ML model\n",
    "* Experiment run: Each trial in an ML experiment\n",
    "* Run artifact: Any file that is associated with an ML run\n",
    "* Experiment metadata: All information related to the experimet\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "816857de",
   "metadata": {},
   "source": [
    "## What is Experiment Tracking?\n",
    "Experiment tracking is the process of keeping track of all the **relevant information** from an **ML experiment**, which includes:\n",
    "* Source code\n",
    "* Environment\n",
    "* Data\n",
    "* Model\n",
    "* Hyperparameters\n",
    "* Metrics\n",
    "* ...\n",
    "\n",
    "What exactly the 'relevant information' is, depends on the specific experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb61e2c5",
   "metadata": {},
   "source": [
    "## Why is Experiment Tracking so important\n",
    "3 main reasons\n",
    "* Reproducability\n",
    "* Organization\n",
    "* Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "492b07c3",
   "metadata": {},
   "source": [
    "## MLflow\n",
    "* Python package, that contains 4 main modules:\n",
    "    * Tracking\n",
    "    * Models\n",
    "    * Model Registry\n",
    "    * Projects\n",
    "* Here we focus on tracking\n",
    "    * MLflow tracking module allows you to organize your experiments into runs, and keep track of\n",
    "        * Parameters\n",
    "        * Metrics\n",
    "        * Metadata\n",
    "        * Artifacts\n",
    "        * Models\n",
    "    * Along with this information, MLflow automatically logs extra information about the run:\n",
    "        * Source code\n",
    "        * Version of the code (git commit)\n",
    "        * Start and end time\n",
    "        * Author\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "440ba3dc",
   "metadata": {},
   "source": [
    "## Getting started\n",
    "* ```pip install mlflow```\n",
    "* typing ```mlflow``` shows the options you have:\n",
    "![mlflow](mlflow.png)\n",
    "* Have a look at the ```ui``` option:\n",
    "    * ```mlflow ui```\n",
    "    * This runs mlflow ui locally\n",
    "    * This gives you access to the experiments via the browser\n",
    "![mlflow](mlflow_ui_terminal.png)\n",
    "![mlflow](mlflow_ui.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19cdfa71",
   "metadata": {},
   "source": [
    "## Example: How to add loging to a Jupyter Notenbook\n",
    "* Create conda environment: ```conda create --name exp-tracking-env python=3.9```\n",
    "* Activate the environment: ```conda activate exp-tracking-env```\n",
    "* Install the requirements: ```pip install -r requirements.txt```\n",
    "* Start mlflow uri: ```mlflow ui --backend-store-uri sqlite:///mlflow.db```\n",
    "    * The option ```backend-store-uri``` here means that we want to store all the artifacts and metadata in an sqlite database\n",
    "    * Copy the notebook mlops-zoomcamp/01-intro/duration-prediction.ipynb\n",
    "    * Create a kernel from the environment: ```conda install -c anaconda ipykernel``` ```python -m ipykernel install --user --name=exp-tracking-env```\n",
    "    * Open the notebook and choose the kernel ```exp-tracking-env```\n",
    "    * Add this to the notebook:\n",
    "    ```import mlflow\n",
    "mlflow.set_tracking_uri(\\\"sqlite:///mlflow.db\\\")\n",
    "mlflow.set_experiment(\\\"nyc-taxi-experiment\\\")```\n",
    "    * To track an experiment add\n",
    "    ```with mlflow.start_run():```\n",
    "      Then everything inside the ```with statement``` will be associated with the current run\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a369b30",
   "metadata": {},
   "source": [
    "## Experiment Tracking with MLflow\n",
    "* Add paramter tuning to the notebook\n",
    "    * Use a second model as example: xgboost\n",
    "    * Use hyperopt for hyperparamtertuning\n",
    "    * Documentation for hyperopt: https://hyperopt.github.io/hyperopt/getting-started/search_spaces\n",
    "* Show how it looks in MLflow\n",
    "    * Different visualisation possibilities: Parallel Coordinates Plot, Scatter Plot, Contour Plot\n",
    "    * Possibility to filter results, e.g. by tags\n",
    "* Select the best one\n",
    "    * One way to select the best model is to sort the results by the metric\n",
    "    * Also consider: training time, model size\n",
    "* Autolog\n",
    "    * Works only with certain frameworks: mlflow.org/docs/latest/tracking.html#automatic-logging\n",
    "    * Enables us to log a lot of information automatically with less code - additional logging maybe necessary\n",
    "    * For xgboost: ```mlflow.xgboost.autolog()```\n",
    "        * This saves automatically a lot of useful paramters and artifacts\n",
    "* Use again the notebook from the previous sesion ```duration-prediction.ipynb```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d87011dc",
   "metadata": {},
   "source": [
    "# Model Management\n",
    "![ml_lifecycle](ml_lifecycle.png)\n",
    "(https://neptune.ai/blog/ml-experiment-tracking)\n",
    "* After deploying the model we may realize that the model needs to be updated\n",
    "* Once we deploy the model, the prediction - monitoring stage starts\n",
    "* As for experiment tracking the way how we manage them can be automized\n",
    "    * E.g. we could manage our models using different folder and filenames. This has several disadvantages:\n",
    "        * it is very error prone\n",
    "        * there is no versioning\n",
    "        * there is no model lineage\n",
    "    * Alternatively we can use mlflow to manage our models\n",
    "        * the most simple way to save a model is: ```mlflow.log_artifact(local_path=\\\"models/lin_reg.bin\\\", artifact_path=\\\"models_pickle/\\\")```, Note: for me this did not work, but only ```mlflow.log_artifact(local_path=\\\"models/lin_reg.bin\\\")```\n",
    "        * better way to save the models:\n",
    "          ```\n",
    "          with mlflow.start_run():\n",
    "            best_params =  {'learning_rate': 0.20905792515510074,\n",
    "                            'max_depth': 7,\n",
    "                            'min_child_weight': 0.5241500975917085,\n",
    "                            'objective': 'reg:squarederror',\n",
    "                            'reg_alpha': 0.13309121698466933,\n",
    "                            'reg_lambda': 0.11277257081373988,\n",
    "                            'seed': 42}\n",
    "            mlflow.log_params(best_params)\n",
    "            booster = xgb.train(\n",
    "                # paramters are passed to xgboost\n",
    "                params=params,\n",
    "                # training on train data\n",
    "                dtrain=train,\n",
    "                # set boosting rounds\n",
    "                num_boost_round=100,\n",
    "                # validation is done on validation dataset\n",
    "                evals=[(valid, 'validation')],\n",
    "                # if model does not improve for 50 methods->stop\n",
    "                early_stopping_rounds=50\n",
    "            )\n",
    "\n",
    "            # make predictions\n",
    "            y_pred = booster.predict(valid)\n",
    "            # calculate error\n",
    "            rmse = mean_squared_error(y_val, y_pred, squared=False)\n",
    "            # log metric\n",
    "            mlflow.log_metric(\\\"rmse\\\", rmse)\n",
    "            # log the model\n",
    "            mlflow.xgboost.log_model(booster, artifact_path=\\\"models_mlflow\\\")\n",
    "        ```\n",
    "        * Note: We have to disable the autolog, else the model will be saved twice: ```mlflow.xgboost.autolog(disable=True)```\n",
    "* Additionally we should log the preprocessor as an artifact\n",
    "* MLflow ui also shows us how to make predictions\n",
    "    * under ```models_mlflow``` examples for spark and pandas dataframe predictions are shown, with the specific model id\n",
    "    * we can load the model by the model id and usee it to make predictions\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6b8cee5",
   "metadata": {},
   "source": [
    "# Model Registry\n",
    "* With MLflow we have a tracking server, were we can track our models, their metrics etc.\n",
    "* At some point we decide that some models are ready for production. We can then store them in the model registry of mlflow\n",
    "* The model registry is not deploying a model, but a list of models that are ready for deployment\n",
    "* To decide for a specific model consider\n",
    "    * metric\n",
    "    * training time\n",
    "    * model size\n",
    "* Once decided to register a model click on \"Register Model\"\n",
    "* In our case we don't have any model registered yet, i.e. we have to create a new one. Call it \"nyc-taxi-regressor\"\n",
    "* Note: For this to work we need to the ```mlflow.set_tracking_uri```\n",
    "* We can then see our registered models under the \"Models\" tab in mlflow\n",
    "* The registered models can be assigned to different stages:\n",
    "    * Staging\n",
    "    * Production\n",
    "    * Archive\n",
    "    * See notebook: 'mlflow.client.ipynb'\n",
    "    * We can interact with the models using ```MlflowClient```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "92fa1f39",
   "metadata": {},
   "source": [
    "## MLflow in Practice\n",
    "* Depending on the scenario different aspects of mlflow are needed and can be used. Sometimes a local storage of the experiments is sufficient in other cases (when working with several people on developing the model) it is important to share the results using a remote server.\n",
    "* Configuring mlflow\n",
    "    * backend store\n",
    "        * local filesystem (if no backend store is set, this is the default)\n",
    "        * SQL Alquemy compatible database (e.g. sqlite)\n",
    "    * artifacts store\n",
    "        * local filesystem (default)\n",
    "        * remote (e.g. S3 bucket)\n",
    "    * tracking server\n",
    "        * no tracking server\n",
    "        * localhost\n",
    "        * remote\n",
    "* Example notebooks of these 3 scenarios can be found in [running-mlflow-examples](running-mlflow-examples)\n",
    "### Benefits of a remote tracking server\n",
    "* The tracking server can easily be deployed on the cloud\n",
    "* Share experiments with other Data Scientists\n",
    "* Collaborate with others to build and deploy models\n",
    "* Give more visibility of the data science efforts\n",
    "### Issues with running a remote (shared) Mlflow server\n",
    "* Security\n",
    "    * restrict access to the server (e.g. access through VPN)\n",
    "* Scalability\n",
    "* Isolation\n",
    "    * define standard for nameing experiments, models and a set of default tags\n",
    "    * restrict access to artifacts (e.g. use s3 buckets living on different AWS accounts)\n",
    "### Mlflow limitations\n",
    "* Authentification & Users: The open source version of mlflow doesn't provide any sort of authentification\n",
    "* Data versioning: to ensure full reproducibility we need to version the data used to train a model. Mlflow doesn't provide a built-in solution for that, but there are a few ways to deal with this limitation\n",
    "* Model/Data Monitoring & Alerting: This is outside of the scope of mlflow, and currently there are no suitable tools for doing this.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c76ccfc7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}