source/examples/rapids-sagemaker-hpo/notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "cloud/aws/sagemaker",
     "workflow/hpo",
     "library/xgboost",
     "library/cuml",
     "library/cupy",
     "library/cudf",
     "library/dask",
     "data-storage/s3",
     "data-format/parquet"
    ]
   },
   "source": [
    "# Deep Dive into Running Hyper Parameter Optimization on AWS SageMaker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Hyper Parameter Optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization) (HPO) improves model quality by searching over hyperparameters, parameters not typically learned during the training process but rather values that control the learning process itself (e.g., model size/capacity). This search can significantly boost model quality relative to default settings and non-expert tuning; however, HPO can take a very long time on a non-accelerated platform. In this notebook, we containerize a RAPIDS workflow and run Bring-Your-Own-Container SageMaker HPO to show how we can overcome the computational complexity of model search. \n",
    "\n",
    "We accelerate HPO in two key ways: \n",
    "* by *scaling within a node* (e.g., multi-GPU where each GPU brings a magnitude higher core count relative to CPUs), and \n",
    "* by *scaling across nodes* and running parallel trials on cloud instances.\n",
    "\n",
    "By combining these two powers HPO experiments that feel unapproachable and may take multiple days on CPU instances can complete in just hours. For example, we find a <span style=\"color:#8735fb; font-size:14pt\"> **12x** </span> speedup in wall clock time (6 hours vs 3+ days) and a <span style=\"color:#8735fb; font-size:14pt\"> **4.5x** </span> reduction in cost when comparing between GPU and CPU [EC2 Spot instances](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html) on 100 XGBoost HPO trials using 10 parallel workers on 10 years of the Airline Dataset (~63M flights) hosted in a S3 bucket. For additional details refer to the <a href='#experiments'>end of the notebook</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all these powerful tools at our disposal, every data scientist should feel empowered to up-level their model before serving it to the world!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../_static/images/examples/rapids-sagemaker-hpo/hpo.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preamble"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get things rolling let's make sure we can query our AWS SageMaker execution role and session as well as our account ID and AWS region."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY   TAG       IMAGE ID   CREATED   SIZE\n"
     ]
    }
   ],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: boto3 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (1.34.119)\n",
      "Requirement already satisfied: botocore<1.35.0,>=1.34.119 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from boto3) (1.34.119)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from boto3) (1.0.1)\n",
      "Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from boto3) (0.10.1)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from botocore<1.35.0,>=1.34.119->boto3) (2.9.0)\n",
      "Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from botocore<1.35.0,>=1.34.119->boto3) (1.26.18)\n",
      "Requirement already satisfied: six>=1.5 in /home/ec2-user/anaconda3/envs/rapids/lib/python3.10/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.35.0,>=1.34.119->boto3) (1.16.0)\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install --upgrade boto3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import sagemaker\n",
    "from helper_functions import (\n",
    "    download_best_model,\n",
    "    new_job_name_from_config,\n",
    "    recommend_instance_type,\n",
    "    summarize_choices,\n",
    "    summarize_hpo_results,\n",
    "    validate_dockerfile,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "execution_role = sagemaker.get_execution_role()\n",
    "session = sagemaker.Session()\n",
    "\n",
    "account = !(aws sts get-caller-identity --query Account --output text)\n",
    "region = !(aws configure get region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(['561241433344'], ['us-west-2'])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "account, region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Key Choices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's go ahead and choose the configuration options for our HPO run.\n",
    "\n",
    "Below are two reference configurations showing a small and a large scale HPO (sized in terms of total experiments/compute). \n",
    "\n",
    "The default values in the notebook are set for the small HPO configuration, however you are welcome to scale them up.\n",
    "\n",
    "> **small HPO**: 1_year, XGBoost, 3 CV folds, singleGPU, max_jobs = 10, max_parallel_jobs = 2\n",
    "\n",
    "> **large HPO**: 10_year, XGBoost, 10 CV folds, multiGPU, max_jobs = 100, max_parallel_jobs = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We offer free hosting for several demo datasets that you can try running HPO with, or alternatively you can bring your own dataset (BYOD). \n",
    "\n",
    "By default we leverage the `Airline` dataset, which is a large public tracker of US domestic flight logs which we offer in various sizes (1 year, 3 year, and 10 year) and in <a href='https://parquet.apache.org/'>Parquet</a> (compressed column storage) format. The machine learning objective with this dataset is to predict whether flights will be more than 15 minutes late arriving to their destination ([dataset link](https://www.transtats.bts.gov/DatabaseInfo.asp?DB_ID=120&DB_URL=), additional details in <a href='#dataset'>Section 1.1</a>). \n",
    "\n",
    "As an alternative we also offer the `NYC Taxi` dataset which captures yellow cab trip details in Ney York in January 2020, stored in <a href='https://en.wikipedia.org/wiki/Comma-separated_values'>CSV </a> format without any compression. The machine learning objective with this dataset is to predict whether a trip had an above average tip (>$2.20).\n",
    "\n",
    "We host the demo datasets in public S3 demo buckets in both the **us-east-1** (N. Virginia) or **us-west-2** (Oregon) regions (i.e., `sagemaker-rapids-hpo-us-east-1`, and `sagemaker-rapids-hpo-us-west-2`). You should run the SageMaker HPO workflow in either of these two regions if you wish to leverage the demo datasets since SageMaker requires that the S3 dataset and the compute you'll be renting are co-located. \n",
    "\n",
    "Lastly, if you plan to use your own dataset refer to the <a href='#byod'>BYOD checklist in the Appendix</a> to help integrate into the workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| dataset | data_bucket | dataset_directory | # samples | storage type | time span |\n",
    "|---|---|---|---|---|---|\n",
    "| Airline Stats Small    | demo    | 1_year   | 6.3M   | Parquet     | 2019         |\n",
    "| Airline Stats Medium   | demo    | 3_year   | 18M    | Parquet     | 2019-2017    |\n",
    "| Airline Stats Large    | demo    | 10_year  | 63M    | Parquet     | 2019-2010    |\n",
    "| NYC Taxi               | demo    | NYC_taxi | 6.3M   | CSV         | 2020 January |\n",
    "| Bring Your Own Dataset | custom  | custom   | custom | Parquet/CSV | custom       |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose dataset S3 bucket and directory\n",
    "data_bucket = \"sagemaker-rapids-hpo-\" + region[0]\n",
    "dataset_directory = \"10_year\"  # '1_year', '3_year', '10_year', 'NYC_taxi'\n",
    "\n",
    "# please choose output bucket for trained model(s)\n",
    "model_output_bucket = session.default_bucket()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "s3_data_input = f\"s3://{data_bucket}/{dataset_directory}\"\n",
    "s3_model_output = f\"s3://{model_output_bucket}/trained-models\"\n",
    "\n",
    "best_hpo_model_local_save_directory = os.getcwd()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From a ML/algorithm perspective, we offer [XGBoost](https://xgboost.readthedocs.io/en/latest/#), [RandomForest](https://docs.rapids.ai/api/cuml/~~~rapids_api_docs_version~~~/cuml_blogs.html#tree-and-forest-models) and [KMeans](https://docs.rapids.ai/api/cuml/~~~rapids_api_docs_version~~~/api.html?highlight=kmeans#cuml.KMeans). You are free to switch between these algorithm choices and everything in the example will continue to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose learning algorithm\n",
    "algorithm_choice = \"XGBoost\"\n",
    "\n",
    "assert algorithm_choice in [\"XGBoost\", \"RandomForest\", \"KMeans\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also optionally increase robustness via reshuffles of the train-test split (i.e., [cross-validation folds](https://scikit-learn.org/stable/modules/cross_validation.html)). Typical values here are between 3 and 10 folds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose cross-validation folds\n",
    "cv_folds = 10\n",
    "\n",
    "assert cv_folds >= 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ML Workflow Compute Choice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We enable the option of running different code variations that unlock increasing amounts of parallelism in the compute workflow. \n",
    "\n",
    "* `singleCPU` = [pandas](https://pandas.pydata.org/) + [sklearn](https://scikit-learn.org/stable/)\n",
    "* `multiCPU`   = [dask](https://dask.org/) + [pandas](https://pandas.pydata.org/) + [sklearn](https://scikit-learn.org/stable/)\n",
    "\n",
    "* <span style=\"color:#8735fb; font-size:14pt\"> RAPIDS </span> `singleGPU` = [cudf](https://github.com/rapidsai/cudf) + [cuml](https://github.com/rapidsai/cuml)\n",
    "* <span style=\"color:#8735fb; font-size:14pt\"> RAPIDS </span> `multiGPU`  = [dask](https://dask.org/) + [cudf](https://github.com/rapidsai/cudf) + [cuml](https://github.com/rapidsai/cuml) \n",
    "\n",
    "All of these code paths are available in the `/workflows` directory for your reference. \n",
    "\n",
    "> **Note that the single-CPU option will leverage multiple cores in the model training portion of the workflow; however, to unlock full parallelism in each stage of the workflow we use [Dask](https://dask.org/).\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose code variant\n",
    "ml_workflow_choice = \"multiGPU\"\n",
    "\n",
    "assert ml_workflow_choice in [\"singleCPU\", \"singleGPU\", \"multiCPU\", \"multiGPU\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Search Ranges and Strategy\n",
    "<a id='strategy-and-param-ranges'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most important choices when running HPO is to choose the bounds of the hyperparameter search process. Below we've set the ranges of the hyperparameters to allow for interesting variation, you are of course welcome to revise these ranges based on domain knowledge especially if you plan to plug in your own dataset. \n",
    "\n",
    "> Note that we support additional algorithm specific parameters (refer to the `parse_hyper_parameter_inputs` function in `HPOConfig.py`), but for demo purposes have limited our choice to the three parameters that overlap between the XGBoost and RandomForest algorithms. For more details see the documentation for [XGBoost parameters](https://xgboost.readthedocs.io/en/latest/parameter.html) and [RandomForest parameters](https://docs.rapids.ai/api/cuml/~~~rapids_api_docs_version~~~/api.html#random-forest). Since KMeans uses different parameters, we adjust accordingly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose HPO search ranges\n",
    "hyperparameter_ranges = {\n",
    "    \"max_depth\": sagemaker.parameter.IntegerParameter(5, 15),\n",
    "    \"n_estimators\": sagemaker.parameter.IntegerParameter(100, 500),\n",
    "    \"max_features\": sagemaker.parameter.ContinuousParameter(0.1, 1.0),\n",
    "}  # see note above for adding additional parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if \"XGBoost\" in algorithm_choice:\n",
    "    # number of trees parameter name difference b/w XGBoost and RandomForest\n",
    "    hyperparameter_ranges[\"num_boost_round\"] = hyperparameter_ranges.pop(\"n_estimators\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if \"KMeans\" in algorithm_choice:\n",
    "    hyperparameter_ranges = {\n",
    "        \"n_clusters\": sagemaker.parameter.IntegerParameter(2, 20),\n",
    "        \"max_iter\": sagemaker.parameter.IntegerParameter(100, 500),\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also choose between a Random and Bayesian search strategy for picking parameter combinations. \n",
    "\n",
    "**Random Search**: Choose a random combination of values from within the ranges for each training job it launches. The choice of hyperparameters doesn't depend on previous results so you can run the maximum number of concurrent workers without affecting the performance of the search. \n",
    "\n",
    "**Bayesian Search**: Make a guess about which hyperparameter combinations are likely to get the best results. After testing the first set of hyperparameter values, hyperparameter tuning uses regression to choose the next set of hyperparameter values to test."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose HPO search strategy\n",
    "search_strategy = \"Random\"\n",
    "\n",
    "assert search_strategy in [\"Random\", \"Bayesian\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Experiment Scale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to decide how may total experiments to run, and how many should run in parallel. Below we have a very conservative number of maximum jobs to run so that you don't accidently spawn large computations when starting out, however for meaningful HPO searches this number should be much higher (e.g., in our experiments we often run 100 max_jobs). Note that you may need to request a [quota limit increase](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html) for additional  `max_parallel_jobs` parallel workers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose total number of HPO experiments[ we have set this number very low to allow for automated CI testing ]\n",
    "max_jobs = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose number of experiments that can run in parallel\n",
    "max_parallel_jobs = 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also set the max duration for an individual job to 24 hours so we don't have run-away compute jobs taking too long."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "max_duration_of_experiment_seconds = 60 * 60 * 24"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Compute Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the dataset size and compute choice we will try to recommend an instance choice*, you are of course welcome to select alternate configurations. \n",
    "> e.g., For the 10_year dataset option, we suggest ml.p3.8xlarge instances (4 GPUs) and ml.m5.24xlarge CPU instances ( we will need upwards of 200GB CPU RAM during model training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "recommended instance type : ml.p3.8xlarge \n",
      "instance details          : 4x GPUs [ V100 ], 64GB GPU memory,  244GB CPU memory\n"
     ]
    }
   ],
   "source": [
    "# we will recommend a compute instance type, feel free to modify\n",
    "instance_type = recommend_instance_type(ml_workflow_choice, dataset_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to choosing our instance type, we can also enable significant savings by leveraging [AWS EC2 Spot Instances](https://aws.amazon.com/ec2/spot/).\n",
    "\n",
    "We **highly recommend** that you set this flag to `True` as it typically leads to 60-70% cost savings. Note, however that you may need to request a [quota limit increase](https://docs.aws.amazon.com/general/latest/gr/sagemaker.html) to enable Spot instances in SageMaker.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# please choose whether spot instances should be used\n",
    "use_spot_instances_flag = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3 data input    =\ts3://sagemaker-rapids-hpo-us-west-2/10_year\n",
      "s3 model output  =\ts3://sagemaker-us-west-2-561241433344/trained-models\n",
      "compute          =\tmultiGPU\n",
      "algorithm        =\tXGBoost, 10 cv-fold\n",
      "instance         =\tml.p3.8xlarge\n",
      "spot instances   =\tTrue\n",
      "hpo strategy     =\tRandom\n",
      "max_experiments  =\t100\n",
      "max_parallel     =\t10\n",
      "max runtime      =\t86400 sec\n"
     ]
    }
   ],
   "source": [
    "summarize_choices(\n",
    "    s3_data_input,\n",
    "    s3_model_output,\n",
    "    ml_workflow_choice,\n",
    "    algorithm_choice,\n",
    "    cv_folds,\n",
    "    instance_type,\n",
    "    use_spot_instances_flag,\n",
    "    search_strategy,\n",
    "    max_jobs,\n",
    "    max_parallel_jobs,\n",
    "    max_duration_of_experiment_seconds,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<span style=\"display: block; text-align: center; color:#8735fb; font-size:30pt\"> **1. ML Workflow** </span>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../_static/images/examples/rapids-sagemaker-hpo/ml_workflow.png' width='800'> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "<a id ='dataset'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default settings for this demo are built to utilize the Airline dataset (Carrier On-Time Performance 1987-2020, available from the [Bureau of Transportation Statistics](https://transtats.bts.gov/Tables.asp?DB_ID=120&DB_Name=Airline%20On-Time%20Performance%20Data&DB_Short_Name=On-Time#)). Below are some additional details about this dataset, we plan to offer a companion notebook that does a deep dive on the data science behind this dataset. Note that if you are using an alternate dataset (e.g., NYC Taxi or BYOData) these details are not relevant.\n",
    "\n",
    "The public dataset contains logs/features about flights in the United States (17 airlines) including:\n",
    "\n",
    "* Locations and distance  ( `Origin`, `Dest`, `Distance` )\n",
    "* Airline / carrier ( `Reporting_Airline` )\n",
    "* Scheduled departure and arrival times ( `CRSDepTime` and `CRSArrTime` )\n",
    "* Actual departure and arrival times ( `DpTime` and `ArrTime` )\n",
    "* Difference between scheduled & actual times ( `ArrDelay` and `DepDelay` )\n",
    "* Binary encoded version of late, aka our target variable ( `ArrDelay15` )\n",
    "\n",
    "Using these features we will build a classifier model to predict whether a flight is going to be more than 15 minutes late on arrival as it prepares to depart."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Python ML Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To build a RAPIDS enabled SageMaker HPO we first need to build a [SageMaker Estimator](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html). An Estimator is a container image that captures all the software needed to run an HPO experiment. The container is augmented with entrypoint code that will be trggered at runtime by each worker. The entrypoint code enables us to write custom models and hook them up to data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to work with SageMaker HPO, the entrypoint logic should parse hyperparameters (supplied by AWS SageMaker), load and split data, build and train a model, score/evaluate the trained model, and emit an output representing the final score for the given hyperparameter setting. We've already built multiple variations of this code.\n",
    "\n",
    "If you would like to make changes by adding your custom model logic feel free to modify the **train.py** and/or the specific workflow files in the `workflows` directory. You are also welcome to uncomment the cells below to load the read/review the code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's switch our working directory to the location of the Estimator entrypoint and library code."
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "source": [
    "rapids-sagemaker-hpo\n",
    "├── notebook.ipynb\n",
    "├── Dockerfile\n",
    "├── entrypoint.sh\n",
    "├── HPOConfig.py\n",
    "├── HPODatasets.py\n",
    "├── MLWorkflow.py\n",
    "├── serve.py\n",
    "├── train.py\n",
    "└── workflows\n",
    "    ├── MLWorkflowMultiCPU.py\n",
    "    ├── MLWorkflowMultiGPU.py\n",
    "    ├── MLWorkflowSingleCPU.py\n",
    "    └── MLWorkflowSingleGPU.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load train.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# %load workflows/MLWorkflowSingleGPU.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Build Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../_static/images/examples/rapids-sagemaker-hpo/estimator.png' width='800'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we've already mentioned, the SageMaker Estimator represents the containerized software stack that AWS SageMaker will replicate to each worker node.\n",
    "\n",
    "The first step to building our Estimator, is to augment a RAPIDS container with our ML Workflow code from above, and push this image to Amazon Elastic Cloud Registry so it is available to SageMaker.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Containerize and Push to ECR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's turn to building our container so that it can integrate with the AWS SageMaker HPO API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our container can either be built on top of the latest RAPIDS [ nightly ] image as a starting layer or the RAPIDS stable image.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "rapids_base_container = \"{{ rapids_container }}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's also decide on the full name of our container."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "image_base = \"sagemaker-rapids-mnmg-100\"\n",
    "image_tag = rapids_base_container.split(\":\")[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ecr_fullname = (\n",
    "    f\"{account[0]}.dkr.ecr.{region[0]}.amazonaws.com/{image_base}:{image_tag}\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'561241433344.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rapids-mnmg-100:24.06a-cuda11.8-py3.10'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ecr_fullname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Write Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write out the Dockerfile to disk, and in a few cells execute the docker build command. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now write our selected RAPDIS image layer as the first FROM statement in the the Dockerfile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"Dockerfile\", \"w\") as dockerfile:\n",
    "    dockerfile.writelines(\n",
    "        f\"FROM {rapids_base_container} \\n\\n\"\n",
    "        f'ENV AWS_DATASET_DIRECTORY=\"{dataset_directory}\"\\n'\n",
    "        f'ENV AWS_ALGORITHM_CHOICE=\"{algorithm_choice}\"\\n'\n",
    "        f'ENV AWS_ML_WORKFLOW_CHOICE=\"{ml_workflow_choice}\"\\n'\n",
    "        f'ENV AWS_CV_FOLDS=\"{cv_folds}\"\\n'\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's append write the remaining pieces of the Dockerfile, namely adding the sagemaker-training-toolkit, flask, dask-ml, and copying our python code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Appending to Dockerfile\n"
     ]
    }
   ],
   "source": [
    "%%writefile -a Dockerfile\n",
    "\n",
    "# ensure printed output/log-messages retain correct order\n",
    "ENV PYTHONUNBUFFERED=True\n",
    "\n",
    "# install a few more dependencies\n",
    "RUN conda install --yes -n base \\\n",
    "        cupy \\\n",
    "        flask \\\n",
    "        protobuf \\\n",
    "        sagemaker\n",
    "\n",
    "# path where SageMaker looks for code when container runs in the cloud\n",
    "ENV CLOUD_PATH=\"/opt/ml/code\"\n",
    "\n",
    "# copy our latest [local] code into the container \n",
    "COPY . $CLOUD_PATH\n",
    "\n",
    "WORKDIR $CLOUD_PATH\n",
    "ENTRYPOINT [\"./entrypoint.sh\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly, let's ensure that our Dockerfile correctly captured our base image selection."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "FROM rapidsai/base:24.06a-cuda11.8-py3.10 \n",
      "\n",
      "ENV AWS_DATASET_DIRECTORY=\"10_year\"\n",
      "ENV AWS_ALGORITHM_CHOICE=\"XGBoost\"\n",
      "ENV AWS_ML_WORKFLOW_CHOICE=\"multiGPU\"\n",
      "ENV AWS_CV_FOLDS=\"10\"\n",
      "\n",
      "# ensure printed output/log-messages retain correct order\n",
      "ENV PYTHONUNBUFFERED=True\n",
      "\n",
      "# install a few more dependencies\n",
      "RUN conda install --yes -n base \\\n",
      "        cupy \\\n",
      "        flask \\\n",
      "        protobuf \\\n",
      "        sagemaker\n",
      "\n",
      "# path where SageMaker looks for code when container runs in the cloud\n",
      "ENV CLOUD_PATH=\"/opt/ml/code\"\n",
      "\n",
      "# copy our latest [local] code into the container \n",
      "COPY . $CLOUD_PATH\n",
      "\n",
      "WORKDIR $CLOUD_PATH\n",
      "ENTRYPOINT [\"./entrypoint.sh\"]\n"
     ]
    }
   ],
   "source": [
    "validate_dockerfile(rapids_base_container)\n",
    "!cat Dockerfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Build and Tag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The build step will be dominated by the download of the RAPIDS image (base layer). If it's already been downloaded the build will take less than 1 minute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24.06a-cuda11.8-py3.10: Pulling from rapidsai/base\n",
      "\n",
      "\u001b[1B8493d397: Pulling fs layer \n",
      "\u001b[1B7ee77381: Pulling fs layer \n",
      "\u001b[1B37f007fd: Pulling fs layer \n",
      "\u001b[1B774ad2ec: Pulling fs layer \n",
      "\u001b[1B22adee62: Pulling fs layer \n",
      "\u001b[1B68414a39: Pulling fs layer \n",
      "\u001b[1B3710f323: Pulling fs layer \n",
      "\u001b[1B8390d4e8: Pulling fs layer \n",
      "\u001b[1Bd0879975: Pulling fs layer \n",
      "\u001b[1B1ab494af: Pulling fs layer \n",
      "\u001b[1B763525ea: Pulling fs layer \n",
      "\u001b[1Bd4a79ee5: Pulling fs layer \n",
      "\u001b[1B4ab83532: Pulling fs layer \n",
      "\u001b[1Bb700ef54: Pulling fs layer \n",
      "\u001b[11B2adee62: Waiting fs layer \n",
      "\u001b[1B5408789c: Pulling fs layer \n",
      "\u001b[1B024a16c8: Pull complete  637B/637B7GBB\u001b[16A\u001b[2K\u001b[15A\u001b[2K\u001b[15A\u001b[2K\u001b[17A\u001b[2K\u001b[14A\u001b[2K\u001b[15A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[12A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[10A\u001b[2K\u001b[9A\u001b[2K\u001b[17A\u001b[2K\u001b[9A\u001b[2K\u001b[11A\u001b[2K\u001b[17A\u001b[2K\u001b[17A\u001b[2K\u001b[11A\u001b[2K\u001b[17A\u001b[2K\u001b[16A\u001b[2K\u001b[16A\u001b[2KExtracting  3.146MB/4.623MB\u001b[16A\u001b[2K\u001b[11A\u001b[2K\u001b[6A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[15A\u001b[2K\u001b[11A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[2A\u001b[2K\u001b[15A\u001b[2K\u001b[2A\u001b[2K\u001b[15A\u001b[2K\u001b[2A\u001b[2K\u001b[13A\u001b[2K\u001b[2A\u001b[2K\u001b[12A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2KDownloading  417.5MB/5.707GB\u001b[11A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[11A\u001b[2K\u001b[2A\u001b[2K\u001b[10A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[2A\u001b[2K\u001b[9A\u001b[2K\u001b[8A\u001b[2K\u001b[7A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[2A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[6A\u001b[2K\u001b[5A\u001b[2K\u001b[2A\u001b[2K\u001b[3A\u001b[2K\u001b[3A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2KExtracting  3.556GB/5.707GB\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2KExtracting   4.58GB/5.707GB\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[2A\u001b[2KExtracting  5.704GB/5.707GB\u001b[2A\u001b[2K\u001b[2A\u001b[2K\u001b[1A\u001b[2K\u001b[1A\u001b[2KDigest: sha256:e1995b699520fbe87a0196e3c24b6fecdd7e45797702e7dca49b4f44da1b23dd\n",
      "Status: Downloaded newer image for rapidsai/base:24.06a-cuda11.8-py3.10\n",
      "docker.io/rapidsai/base:24.06a-cuda11.8-py3.10\n"
     ]
    }
   ],
   "source": [
    "!docker pull $rapids_base_container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY      TAG                      IMAGE ID       CREATED      SIZE\n",
      "rapidsai/base   24.06a-cuda11.8-py3.10   a80bdce0d796   2 days ago   11.3GB\n"
     ]
    }
   ],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sending build context to Docker daemon  455.2kB\n",
      "Step 1/11 : FROM rapidsai/base:24.06a-cuda11.8-py3.10\n",
      " ---> a80bdce0d796\n",
      "Step 2/11 : ENV AWS_DATASET_DIRECTORY=\"10_year\"\n",
      " ---> Running in 6cedf114d041\n",
      "Removing intermediate container 6cedf114d041\n",
      " ---> 6d41b5d4065d\n",
      "Step 3/11 : ENV AWS_ALGORITHM_CHOICE=\"XGBoost\"\n",
      " ---> Running in d419939bfbf0\n",
      "Removing intermediate container d419939bfbf0\n",
      " ---> 8795a2713d5d\n",
      "Step 4/11 : ENV AWS_ML_WORKFLOW_CHOICE=\"multiGPU\"\n",
      " ---> Running in 3cae981f7cab\n",
      "Removing intermediate container 3cae981f7cab\n",
      " ---> 0678993d72a9\n",
      "Step 5/11 : ENV AWS_CV_FOLDS=\"10\"\n",
      " ---> Running in 150f81bcea62\n",
      "Removing intermediate container 150f81bcea62\n",
      " ---> 7aaa519ee897\n",
      "Step 6/11 : ENV PYTHONUNBUFFERED=True\n",
      " ---> Running in 3a4a2d78589c\n",
      "Removing intermediate container 3a4a2d78589c\n",
      " ---> 79a2ad0e4dcc\n",
      "Step 7/11 : RUN conda install --yes -n base         cupy         flask         protobuf         sagemaker\n",
      " ---> Running in 0c9a24d11804\n",
      "Channels:\n",
      " - rapidsai-nightly\n",
      " - dask/label/dev\n",
      " - pytorch\n",
      " - conda-forge\n",
      " - nvidia\n",
      "Platform: linux-64\n",
      "Collecting package metadata (repodata.json): ...working... done\n",
      "Solving environment: ...working... done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /opt/conda\n",
      "\n",
      "  added / updated specs:\n",
      "    - cupy\n",
      "    - flask\n",
      "    - protobuf\n",
      "    - sagemaker\n",
      "\n",
      "\n",
      "The following packages will be downloaded:\n",
      "\n",
      "    package                    |            build\n",
      "    ---------------------------|-----------------\n",
      "    blinker-1.8.2              |     pyhd8ed1ab_0          14 KB  conda-forge\n",
      "    boto3-1.34.119             |     pyhd8ed1ab_0          79 KB  conda-forge\n",
      "    botocore-1.34.119          |pyge310_1234567_0         6.8 MB  conda-forge\n",
      "    dill-0.3.8                 |     pyhd8ed1ab_0          86 KB  conda-forge\n",
      "    flask-3.0.3                |     pyhd8ed1ab_0          79 KB  conda-forge\n",
      "    google-pasta-0.2.0         |     pyh8c360ce_0          42 KB  conda-forge\n",
      "    itsdangerous-2.2.0         |     pyhd8ed1ab_0          19 KB  conda-forge\n",
      "    jmespath-1.0.1             |     pyhd8ed1ab_0          21 KB  conda-forge\n",
      "    multiprocess-0.70.16       |  py310h2372a71_0         238 KB  conda-forge\n",
      "    openssl-3.3.1              |       h4ab18f5_0         2.8 MB  conda-forge\n",
      "    pathos-0.3.2               |     pyhd8ed1ab_1          52 KB  conda-forge\n",
      "    pox-0.3.4                  |     pyhd8ed1ab_0          26 KB  conda-forge\n",
      "    ppft-1.7.6.8               |     pyhd8ed1ab_0          33 KB  conda-forge\n",
      "    protobuf-4.25.3            |  py310ha8c1f0e_0         325 KB  conda-forge\n",
      "    protobuf3-to-dict-0.1.5    |  py310hff52083_8          14 KB  conda-forge\n",
      "    s3transfer-0.10.1          |     pyhd8ed1ab_0          61 KB  conda-forge\n",
      "    sagemaker-2.75.1           |     pyhd8ed1ab_0         377 KB  conda-forge\n",
      "    smdebug-rulesconfig-1.0.1  |     pyhd3deb0d_1          20 KB  conda-forge\n",
      "    werkzeug-3.0.3             |     pyhd8ed1ab_0         237 KB  conda-forge\n",
      "    ------------------------------------------------------------\n",
      "                                           Total:        11.2 MB\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  blinker            conda-forge/noarch::blinker-1.8.2-pyhd8ed1ab_0 \n",
      "  boto3              conda-forge/noarch::boto3-1.34.119-pyhd8ed1ab_0 \n",
      "  botocore           conda-forge/noarch::botocore-1.34.119-pyge310_1234567_0 \n",
      "  dill               conda-forge/noarch::dill-0.3.8-pyhd8ed1ab_0 \n",
      "  flask              conda-forge/noarch::flask-3.0.3-pyhd8ed1ab_0 \n",
      "  google-pasta       conda-forge/noarch::google-pasta-0.2.0-pyh8c360ce_0 \n",
      "  itsdangerous       conda-forge/noarch::itsdangerous-2.2.0-pyhd8ed1ab_0 \n",
      "  jmespath           conda-forge/noarch::jmespath-1.0.1-pyhd8ed1ab_0 \n",
      "  multiprocess       conda-forge/linux-64::multiprocess-0.70.16-py310h2372a71_0 \n",
      "  pathos             conda-forge/noarch::pathos-0.3.2-pyhd8ed1ab_1 \n",
      "  pox                conda-forge/noarch::pox-0.3.4-pyhd8ed1ab_0 \n",
      "  ppft               conda-forge/noarch::ppft-1.7.6.8-pyhd8ed1ab_0 \n",
      "  protobuf           conda-forge/linux-64::protobuf-4.25.3-py310ha8c1f0e_0 \n",
      "  protobuf3-to-dict  conda-forge/linux-64::protobuf3-to-dict-0.1.5-py310hff52083_8 \n",
      "  s3transfer         conda-forge/noarch::s3transfer-0.10.1-pyhd8ed1ab_0 \n",
      "  sagemaker          conda-forge/noarch::sagemaker-2.75.1-pyhd8ed1ab_0 \n",
      "  smdebug-rulesconf~ conda-forge/noarch::smdebug-rulesconfig-1.0.1-pyhd3deb0d_1 \n",
      "  werkzeug           conda-forge/noarch::werkzeug-3.0.3-pyhd8ed1ab_0 \n",
      "\n",
      "The following packages will be UPDATED:\n",
      "\n",
      "  openssl                                  3.3.0-h4ab18f5_3 --> 3.3.1-h4ab18f5_0 \n",
      "\n",
      "\n",
      "\n",
      "Downloading and Extracting Packages: ...working... done\n",
      "Preparing transaction: ...working... done\n",
      "Verifying transaction: ...working... done\n",
      "Executing transaction: ...working... done\n",
      "Removing intermediate container 0c9a24d11804\n",
      " ---> 14a9c592c437\n",
      "Step 8/11 : ENV CLOUD_PATH=\"/opt/ml/code\"\n",
      " ---> Running in bce7a9615822\n",
      "Removing intermediate container bce7a9615822\n",
      " ---> ae3d6adb9f11\n",
      "Step 9/11 : COPY . $CLOUD_PATH\n",
      " ---> 6fdd122a0b80\n",
      "Step 10/11 : WORKDIR $CLOUD_PATH\n",
      " ---> Running in f259ffd20090\n",
      "Removing intermediate container f259ffd20090\n",
      " ---> b7c9af873fa8\n",
      "Step 11/11 : ENTRYPOINT [\"./entrypoint.sh\"]\n",
      " ---> Running in 22cd3b9875fb\n",
      "Removing intermediate container 22cd3b9875fb\n",
      " ---> 5eb1ea40d589\n",
      "Successfully built 5eb1ea40d589\n",
      "Successfully tagged 561241433344.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rapids-mnmg-100:24.06a-cuda11.8-py3.10\n",
      "CPU times: user 977 ms, sys: 240 ms, total: 1.22 s\n",
      "Wall time: 1min 15s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "!docker build -t $ecr_fullname ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY                                                               TAG                      IMAGE ID       CREATED         SIZE\n",
      "561241433344.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rapids-mnmg-100   24.06a-cuda11.8-py3.10   5eb1ea40d589   2 minutes ago   12GB\n",
      "rapidsai/base                                                            24.06a-cuda11.8-py3.10   a80bdce0d796   2 days ago      11.3GB\n"
     ]
    }
   ],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Publish to Elastic Cloud Registry (ECR)\n",
    "\n",
    "Now that we've built and tagged our container its time to push it to Amazon's container registry (ECR). Once in ECR, AWS SageMaker will be able to leverage our image to build Estimators and run experiments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Docker Login to ECR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docker_login_str = !(aws ecr get-login --region {region[0]} --no-include-email)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING! Using --password via the CLI is insecure. Use --password-stdin.\n",
      "WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.\n",
      "Configure a credential helper to remove this warning. See\n",
      "https://docs.docker.com/engine/reference/commandline/login/#credentials-store\n",
      "\n",
      "Login Succeeded\n"
     ]
    }
   ],
   "source": [
    "!{docker_login_str[0]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create ECR repository [ if it doesn't already exist]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "repository_query = !(aws ecr describe-repositories --repository-names $image_base)\n",
    "if repository_query[0] == \"\":\n",
    "    !(aws ecr create-repository --repository-name $image_base)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now actually push the container to ECR\n",
    "> Note the first push to ECR may take some time (hopefully less than 10 minutes)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The push refers to repository [561241433344.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rapids-mnmg-100]\n",
      "\n",
      "\u001b[1B53ae5466: Preparing \n",
      "\u001b[1B7b5ba20f: Preparing \n",
      "\u001b[1Bbdce5066: Preparing \n",
      "\u001b[1B923ec1b3: Preparing \n",
      "\u001b[1B3fcfb3d4: Preparing \n",
      "\u001b[1Bbf18a086: Preparing \n",
      "\u001b[1Bf3ff1008: Preparing \n",
      "\u001b[1Bb6fb91b8: Preparing \n",
      "\u001b[1B7bf1eb99: Preparing \n",
      "\u001b[1B264186e1: Preparing \n",
      "\u001b[1B7d7711e0: Preparing \n",
      "\u001b[1Bee96f292: Preparing \n",
      "\u001b[1Be2a80b3f: Preparing \n",
      "\u001b[1B0a873d7a: Preparing \n",
      "\u001b[1Bbcc60d01: Preparing \n",
      "\u001b[1B1dcee623: Preparing \n",
      "\u001b[1B9a46b795: Preparing \n",
      "\u001b[13Bf18a086: Waiting g \n",
      "\u001b[3B9a46b795: Layer already exists \u001b[14A\u001b[2K\u001b[8A\u001b[2K\u001b[4A\u001b[2K24.06a-cuda11.8-py3.10: digest: sha256:ad469ed961b5a7cee6ff7d71069ead5d2bde9906097298075d320d3d5c305d29 size: 4299\n"
     ]
    }
   ],
   "source": [
    "!docker push $ecr_fullname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having built our container [ +custom logic] and pushed it to ECR, we can finally compile all of efforts into an Estimator instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "REPOSITORY                                                               TAG                      IMAGE ID       CREATED         SIZE\n",
      "561241433344.dkr.ecr.us-west-2.amazonaws.com/sagemaker-rapids-mnmg-100   24.06a-cuda11.8-py3.10   5eb1ea40d589   4 minutes ago   12GB\n",
      "rapidsai/base                                                            24.06a-cuda11.8-py3.10   a80bdce0d796   2 days ago      11.3GB\n"
     ]
    }
   ],
   "source": [
    "!docker images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# 'volume_size' - EBS volume size in GB, default = 30\n",
    "estimator_params = {\n",
    "    \"image_uri\": ecr_fullname,\n",
    "    \"role\": execution_role,\n",
    "    \"instance_type\": instance_type,\n",
    "    \"instance_count\": 2,\n",
    "    \"input_mode\": \"File\",\n",
    "    \"output_path\": s3_model_output,\n",
    "    \"use_spot_instances\": use_spot_instances_flag,\n",
    "    \"max_run\": max_duration_of_experiment_seconds,  # 24 hours\n",
    "    \"sagemaker_session\": session,\n",
    "}\n",
    "\n",
    "if use_spot_instances_flag:\n",
    "    estimator_params.update({\"max_wait\": max_duration_of_experiment_seconds + 1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator = sagemaker.estimator.Estimator(**estimator_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Test Estimator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are ready to test by asking SageMaker to run the BYOContainer logic inside our Estimator. This is a useful step if you've made changes to your custom logic and are interested in making sure everything works before launching a large HPO search. \n",
    "\n",
    "> Note: This verification step will use the default hyperparameter values declared in our custom train code, as SageMaker HPO will not be orchestrating a search for this single run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3 data input    =\ts3://sagemaker-rapids-hpo-us-west-2/10_year\n",
      "s3 model output  =\ts3://sagemaker-us-west-2-561241433344/trained-models\n",
      "compute          =\tmultiGPU\n",
      "algorithm        =\tXGBoost, 10 cv-fold\n",
      "instance         =\tml.p3.8xlarge\n",
      "spot instances   =\tTrue\n",
      "hpo strategy     =\tRandom\n",
      "max_experiments  =\t100\n",
      "max_parallel     =\t10\n",
      "max runtime      =\t86400 sec\n"
     ]
    }
   ],
   "source": [
    "summarize_choices(\n",
    "    s3_data_input,\n",
    "    s3_model_output,\n",
    "    ml_workflow_choice,\n",
    "    algorithm_choice,\n",
    "    cv_folds,\n",
    "    instance_type,\n",
    "    use_spot_instances_flag,\n",
    "    search_strategy,\n",
    "    max_jobs,\n",
    "    max_parallel_jobs,\n",
    "    max_duration_of_experiment_seconds,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generated job name : air-mGPU-XGB-10cv-31d03d8b015bfc\n",
      "\n"
     ]
    }
   ],
   "source": [
    "job_name = new_job_name_from_config(\n",
    "    dataset_directory,\n",
    "    region,\n",
    "    ml_workflow_choice,\n",
    "    algorithm_choice,\n",
    "    cv_folds,\n",
    "    instance_type,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "estimator.fit(inputs=s3_data_input, job_name=job_name.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run HPO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a working SageMaker Estimator in hand, the hardest part is behind us. In the key choices section we <a href='#strategy-and-param-ranges'>already defined our search strategy and hyperparameter ranges</a>, so all that remains is to choose a metric to evaluate performance on. For more documentation check out the AWS SageMaker [Hyperparameter Tuner documentation](https://sagemaker.readthedocs.io/en/stable/tuner.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../_static/images/examples/rapids-sagemaker-hpo/run_hpo.png'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Metric"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only focus on a single metric, which we call 'final-score', that captures the accuracy of our model on the test data unseen during training. You are of course welcome to add aditional metrics, see [AWS SageMaker documentation on Metrics](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning-define-metrics.html). When defining a metric we provide a regular expression (i.e., string parsing rule) to extract the key metric from the output of each Estimator/worker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_definitions = [{\"Name\": \"final-score\", \"Regex\": \"final-score: (.*);\"}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = \"final-score\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Tuner"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally we put all of the elements we've been building up together into a HyperparameterTuner declaration. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "hpo = sagemaker.tuner.HyperparameterTuner(\n",
    "    estimator=estimator,\n",
    "    metric_definitions=metric_definitions,\n",
    "    objective_metric_name=objective_metric_name,\n",
    "    objective_type=\"Maximize\",\n",
    "    hyperparameter_ranges=hyperparameter_ranges,\n",
    "    strategy=search_strategy,\n",
    "    max_jobs=max_jobs,\n",
    "    max_parallel_jobs=max_parallel_jobs,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run HPO"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3 data input    =\ts3://sagemaker-rapids-hpo-us-west-2/10_year\n",
      "s3 model output  =\ts3://sagemaker-us-west-2-561241433344/trained-models\n",
      "compute          =\tmultiGPU\n",
      "algorithm        =\tXGBoost, 10 cv-fold\n",
      "instance         =\tml.p3.8xlarge\n",
      "spot instances   =\tTrue\n",
      "hpo strategy     =\tRandom\n",
      "max_experiments  =\t100\n",
      "max_parallel     =\t10\n",
      "max runtime      =\t86400 sec\n"
     ]
    }
   ],
   "source": [
    "summarize_choices(\n",
    "    s3_data_input,\n",
    "    s3_model_output,\n",
    "    ml_workflow_choice,\n",
    "    algorithm_choice,\n",
    "    cv_folds,\n",
    "    instance_type,\n",
    "    use_spot_instances_flag,\n",
    "    search_strategy,\n",
    "    max_jobs,\n",
    "    max_parallel_jobs,\n",
    "    max_duration_of_experiment_seconds,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's be sure we take a moment to confirm before launching all of our HPO experiments. Depending on your configuration options running this cell can kick off a massive amount of computation!\n",
    "> Once this process begins, we recommend that you use the SageMaker UI to keep track of the <a href='../../_static/images/examples/rapids-sagemaker-hpo/gpu_hpo_100x10.png'>health of the HPO process and the individual workers</a>."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "# tuning_job_name = new_job_name_from_config(dataset_directory, region, ml_workflow_choice,\n",
    "#                                            algorithm_choice, cv_folds,\n",
    "# #                                            instance_type)\n",
    "# hpo.fit( inputs=s3_data_input,\n",
    "#          job_name=tuning_job_name,\n",
    "#          wait=True,\n",
    "#          logs='All')\n",
    "\n",
    "# hpo.wait()  # block until the .fit call above is completed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results and Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once your job is complete there are multiple ways to analyze the results.\n",
    "Below we display the performance of the best job, as well printing each HPO trial/job as a row of a dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuning_job_name = \"air-mGPU-XGB-10cv-527fd372fa4d8d\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best score: 0.9203665256500244\n",
      "best params: {'max_depth': '7', 'max_features': '0.29751893065195945', 'num_boost_round': '346'}\n",
      "best job-name: air-mGPU-XGB-10cv-527fd372fa4d8d-042-ed1ff13b\n"
     ]
    }
   ],
   "source": [
    "hpo_results = summarize_hpo_results(tuning_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>max_depth</th>\n",
       "      <th>max_features</th>\n",
       "      <th>num_boost_round</th>\n",
       "      <th>TrainingJobName</th>\n",
       "      <th>TrainingJobStatus</th>\n",
       "      <th>FinalObjectiveValue</th>\n",
       "      <th>TrainingStartTime</th>\n",
       "      <th>TrainingEndTime</th>\n",
       "      <th>TrainingElapsedTimeSeconds</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.715196</td>\n",
       "      <td>116.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-100-c04c691b</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920362</td>\n",
       "      <td>2023-01-23 21:01:38+00:00</td>\n",
       "      <td>2023-01-23 21:06:21+00:00</td>\n",
       "      <td>283.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12.0</td>\n",
       "      <td>0.855974</td>\n",
       "      <td>243.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-099-97d44628</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920355</td>\n",
       "      <td>2023-01-23 21:17:56+00:00</td>\n",
       "      <td>2023-01-23 21:22:34+00:00</td>\n",
       "      <td>278.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11.0</td>\n",
       "      <td>0.549247</td>\n",
       "      <td>395.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-098-e74f483f</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920356</td>\n",
       "      <td>2023-01-23 20:56:06+00:00</td>\n",
       "      <td>2023-01-23 21:00:44+00:00</td>\n",
       "      <td>278.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>7.0</td>\n",
       "      <td>0.882803</td>\n",
       "      <td>179.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-097-50755cd6</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920356</td>\n",
       "      <td>2023-01-23 20:54:35+00:00</td>\n",
       "      <td>2023-01-23 20:59:13+00:00</td>\n",
       "      <td>278.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.416939</td>\n",
       "      <td>267.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-096-5c95eb2f</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920355</td>\n",
       "      <td>2023-01-23 20:51:24+00:00</td>\n",
       "      <td>2023-01-23 20:56:02+00:00</td>\n",
       "      <td>278.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.426204</td>\n",
       "      <td>330.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-005-7b755a81</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920355</td>\n",
       "      <td>2023-01-23 18:48:35+00:00</td>\n",
       "      <td>2023-01-23 18:53:48+00:00</td>\n",
       "      <td>313.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.283752</td>\n",
       "      <td>256.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-004-e4d086fb</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920356</td>\n",
       "      <td>2023-01-23 18:48:34+00:00</td>\n",
       "      <td>2023-01-23 18:53:47+00:00</td>\n",
       "      <td>313.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.137874</td>\n",
       "      <td>377.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-003-89cd8506</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920355</td>\n",
       "      <td>2023-01-23 18:48:31+00:00</td>\n",
       "      <td>2023-01-23 18:53:44+00:00</td>\n",
       "      <td>313.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>15.0</td>\n",
       "      <td>0.934718</td>\n",
       "      <td>365.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-002-caf8f6c3</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920360</td>\n",
       "      <td>2023-01-23 18:48:25+00:00</td>\n",
       "      <td>2023-01-23 18:53:48+00:00</td>\n",
       "      <td>323.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>15.0</td>\n",
       "      <td>0.356588</td>\n",
       "      <td>460.0</td>\n",
       "      <td>air-mGPU-XGB-10cv-527fd372fa4d8d-001-e8a6d247</td>\n",
       "      <td>Completed</td>\n",
       "      <td>0.920356</td>\n",
       "      <td>2023-01-23 18:48:29+00:00</td>\n",
       "      <td>2023-01-23 18:53:47+00:00</td>\n",
       "      <td>318.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    max_depth  max_features  num_boost_round  \\\n",
       "0         5.0      0.715196            116.0   \n",
       "1        12.0      0.855974            243.0   \n",
       "2        11.0      0.549247            395.0   \n",
       "3         7.0      0.882803            179.0   \n",
       "4         8.0      0.416939            267.0   \n",
       "..        ...           ...              ...   \n",
       "95        5.0      0.426204            330.0   \n",
       "96        5.0      0.283752            256.0   \n",
       "97        5.0      0.137874            377.0   \n",
       "98       15.0      0.934718            365.0   \n",
       "99       15.0      0.356588            460.0   \n",
       "\n",
       "                                  TrainingJobName TrainingJobStatus  \\\n",
       "0   air-mGPU-XGB-10cv-527fd372fa4d8d-100-c04c691b         Completed   \n",
       "1   air-mGPU-XGB-10cv-527fd372fa4d8d-099-97d44628         Completed   \n",
       "2   air-mGPU-XGB-10cv-527fd372fa4d8d-098-e74f483f         Completed   \n",
       "3   air-mGPU-XGB-10cv-527fd372fa4d8d-097-50755cd6         Completed   \n",
       "4   air-mGPU-XGB-10cv-527fd372fa4d8d-096-5c95eb2f         Completed   \n",
       "..                                            ...               ...   \n",
       "95  air-mGPU-XGB-10cv-527fd372fa4d8d-005-7b755a81         Completed   \n",
       "96  air-mGPU-XGB-10cv-527fd372fa4d8d-004-e4d086fb         Completed   \n",
       "97  air-mGPU-XGB-10cv-527fd372fa4d8d-003-89cd8506         Completed   \n",
       "98  air-mGPU-XGB-10cv-527fd372fa4d8d-002-caf8f6c3         Completed   \n",
       "99  air-mGPU-XGB-10cv-527fd372fa4d8d-001-e8a6d247         Completed   \n",
       "\n",
       "    FinalObjectiveValue         TrainingStartTime           TrainingEndTime  \\\n",
       "0              0.920362 2023-01-23 21:01:38+00:00 2023-01-23 21:06:21+00:00   \n",
       "1              0.920355 2023-01-23 21:17:56+00:00 2023-01-23 21:22:34+00:00   \n",
       "2              0.920356 2023-01-23 20:56:06+00:00 2023-01-23 21:00:44+00:00   \n",
       "3              0.920356 2023-01-23 20:54:35+00:00 2023-01-23 20:59:13+00:00   \n",
       "4              0.920355 2023-01-23 20:51:24+00:00 2023-01-23 20:56:02+00:00   \n",
       "..                  ...                       ...                       ...   \n",
       "95             0.920355 2023-01-23 18:48:35+00:00 2023-01-23 18:53:48+00:00   \n",
       "96             0.920356 2023-01-23 18:48:34+00:00 2023-01-23 18:53:47+00:00   \n",
       "97             0.920355 2023-01-23 18:48:31+00:00 2023-01-23 18:53:44+00:00   \n",
       "98             0.920360 2023-01-23 18:48:25+00:00 2023-01-23 18:53:48+00:00   \n",
       "99             0.920356 2023-01-23 18:48:29+00:00 2023-01-23 18:53:47+00:00   \n",
       "\n",
       "    TrainingElapsedTimeSeconds  \n",
       "0                        283.0  \n",
       "1                        278.0  \n",
       "2                        278.0  \n",
       "3                        278.0  \n",
       "4                        278.0  \n",
       "..                         ...  \n",
       "95                       313.0  \n",
       "96                       313.0  \n",
       "97                       313.0  \n",
       "98                       323.0  \n",
       "99                       318.0  \n",
       "\n",
       "[100 rows x 9 columns]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name).dataframe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " For a more in depth look at the HPO process we invite you to check out the <a href='https://github.com/awslabs/amazon-sagemaker-examples/tree/master/hyperparameter_tuning/analyze_results'>HPO_Analyze_TuningJob_Results.ipynb</a> notebook which shows how we can explore interesting things like the <a href='../../_static/images/examples/rapids-sagemaker-hpo/results_analysis.png'>impact of each individual hyperparameter on the performance metric</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Getting the Best Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next let's download the best trained model from our HPO runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:botocore.credentials:Found credentials from IAM Role: BaseNotebookInstanceEc2InstanceRole\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Successfully downloaded best model\n",
      "> filename: /home/ec2-user/SageMaker/cloud-ml-examples/aws/best_model.tar.gz\n",
      "> local directory : /home/ec2-user/SageMaker/cloud-ml-examples/aws\n",
      "\n",
      "full S3 path : s3://sagemaker-us-west-2-561241433344/trained-models/air-mGPU-XGB-10cv-527fd372fa4d8d-042-ed1ff13b/output/model.tar.gz\n"
     ]
    }
   ],
   "source": [
    "local_filename, s3_path_to_best_model = download_best_model(\n",
    "    model_output_bucket,\n",
    "    s3_model_output,\n",
    "    hpo_results,\n",
    "    best_hpo_model_local_save_directory,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Serving\n",
    "\n",
    "With your best model in hand, you can now move on to [serving this model on SageMaker](https://docs.aws.amazon.com/sagemaker/latest/dg/how-it-works-deployment.html). \n",
    "\n",
    "In the example below we show you how to build a [RealTimePredictor](https://sagemaker.readthedocs.io/en/stable/api/inference/predictors.html) using the best model found during the HPO search. We will add a lightweight Flask server to our RAPIDS Estimator (a.k.a., container) which will handle the incoming requests and pass them along to the trained model for inference. If you are curious about how this works under the hood check out the [Use Your Own Inference Server](https://docs.aws.amazon.com/sagemaker/latest/dg/your-algorithms-inference-code.html) documentation and reference the code in `serve.py`.\n",
    "\n",
    "If you are interested in additional serving options (e.g., large batch with batch-transform), we plan to add a companion notebook that will provide additional details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GPU serving"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint_model = sagemaker.model.Model(\n",
    "    image_uri=ecr_fullname, role=execution_role, model_data=s3_path_to_best_model\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'561241433344.dkr.ecr.us-west-2.amazonaws.com/rapids-sagemaker-mnmg-100:22.12-cuda11.5-runtime-ubuntu18.04-py3.9'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ecr_fullname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Kick off an instance for prediction [ recommend 'ml.g4dn.2xlarge' ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating model with name: rapids-sagemaker-mnmg-100-2023-01-23-22-24-22-008\n",
      "INFO:sagemaker:Creating endpoint-config with name rapids-sagemaker-mnmg-100-2023-01-23-22-24-22-498\n",
      "INFO:sagemaker:Creating endpoint with name rapids-sagemaker-mnmg-100-2023-01-23-22-24-22-498\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------!"
     ]
    }
   ],
   "source": [
    "DEMO_SERVING_FLAG = True\n",
    "\n",
    "if DEMO_SERVING_FLAG:\n",
    "    endpoint_model.deploy(\n",
    "        initial_instance_count=1, instance_type=\"ml.g4dn.2xlarge\"\n",
    "    )  #'ml.p3.2xlarge'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Perform the prediction and return the result(s)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below we've compiled examples to sanity test the trained model performance on the Airline dataset.\n",
    "> The first example is from a 2019 flight that departed nine minutes early, \n",
    "\n",
    "> The second example is from a 2018 flight that was more than two hours late to depart.\n",
    "\n",
    "When we run these samples we expect to see **b'[0.0, 1.0]** as the printed result.\n",
    "\n",
    "We encourage you to modify the queries below especially if you plug in your own dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'[0.0, 1.0]'\n"
     ]
    }
   ],
   "source": [
    "if DEMO_SERVING_FLAG:\n",
    "    predictor = sagemaker.predictor.Predictor(\n",
    "        endpoint_name=str(endpoint_model.endpoint_name), sagemaker_session=session\n",
    "    )\n",
    "\n",
    "    if dataset_directory in [\"1_year\", \"3_year\", \"10_year\"]:\n",
    "        on_time_example = [\n",
    "            2019.0,\n",
    "            4.0,\n",
    "            12.0,\n",
    "            2.0,\n",
    "            3647.0,\n",
    "            20452.0,\n",
    "            30977.0,\n",
    "            33244.0,\n",
    "            1943.0,\n",
    "            -9.0,\n",
    "            0.0,\n",
    "            75.0,\n",
    "            491.0,\n",
    "        ]  # 9 minutes early departure\n",
    "        late_example = [\n",
    "            2018.0,\n",
    "            3.0,\n",
    "            9.0,\n",
    "            5.0,\n",
    "            2279.0,\n",
    "            20409.0,\n",
    "            30721.0,\n",
    "            31703.0,\n",
    "            733.0,\n",
    "            123.0,\n",
    "            1.0,\n",
    "            61.0,\n",
    "            200.0,\n",
    "        ]\n",
    "        example_payload = str(list([on_time_example, late_example]))\n",
    "    else:\n",
    "        example_payload = \"\"  # fill in a sample payload\n",
    "\n",
    "    result = predictor.predict(example_payload)\n",
    "    print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we are finished with the serving example, we should be sure to clean up and delete the endpoint. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if DEMO_SERVING_FLAG:\n",
    "\n",
    "#     predictor.delete_endpoint()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've now successfully built a RAPIDS ML workflow, containerized it (as a SageMaker Estimator), and launched a set of HPO experiments to find the best hyperparamters for our model.\n",
    "\n",
    "If you are curious to go further, we invite you to plug in your own dataset and tweak the configuration settings to find your champion model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**HPO Experiment Details**\n",
    "<a id='experiments'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned in the introduction we find a <span style=\"color:#8735fb; font-size:14pt\"> **12X** </span> speedup in wall clock time and a <span style=\"color:#8735fb; font-size:14pt\"> **4.5x** </span> reduction in cost when comparing between GPU and CPU instances on 100 HPO trials using 10 parallel workers on 10 years of the Airline Dataset (~63M flights). In these experiments we used the XGBoost algorithm with the multi-GPU vs multi-CPU Dask cluster and 10 cross validaiton folds. Below we offer a table with additional details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='../../_static/images/examples/rapids-sagemaker-hpo/results.png' width='70%'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case of the CPU runs, 12 jobs were stopped since they exceeded the 24 hour limit we set. <a href='../../_static/images/examples/rapids-sagemaker-hpo/cpu_hpo_100x10.png'>CPU Job Summary Image</a>\n",
    "\n",
    "In the case of the GPU runs, no jobs were stopped. <a href='../../_static/images/examples/rapids-sagemaker-hpo/gpu_hpo_100x10.png'>GPU Job Summary Image</a>\n",
    "\n",
    "Note that in both cases 1 job failed because a spot instance was terminated. But 1 failed job out of 100 is a minimal tradeoff for the significant cost savings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appendix\n",
    "\n",
    "### Bring Your Own Dataset Checklist\n",
    "<a id ='byod'></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you plan to use your own dataset (BYOD) here is a checklist to help you integrate into the workflow:\n",
    "\n",
    "> - [ ] Dataset should be in either CSV or Parquet format.\n",
    "> - [ ] Dataset is already pre-processed (and all feature-engineering is done).\n",
    "> - [ ] Dataset is uploaded to S3 and `data_bucket` and `dataset_directory` have been set to the location of your data.\n",
    "> - [ ] Dataset feature and target columns have been enumerated in `/HPODataset.py`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rapids References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> [More Cloud Deployment Workflow Examples](https://docs.rapids.ai/deployment/stable/examples/)\n",
    "\n",
    "> [RAPIDS HPO](https://rapids.ai/hpo)\n",
    "\n",
    "> [cuML Documentation](https://docs.rapids.ai/api/cuml/~~~rapids_api_docs_version~~~/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SageMaker References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> [SageMaker Training Toolkit](https://github.com/aws/sagemaker-training-toolkit)\n",
    "\n",
    "> [Estimator Parameters](https://sagemaker.readthedocs.io/en/stable/api/training/estimators.html)\n",
    "\n",
    "> Spot Instances [docs](https://docs.aws.amazon.com/sagemaker/latest/dg/model-managed-spot-training.html), and [blog](https://aws.amazon.com/blogs/aws/managed-spot-training-save-up-to-90-on-your-amazon-sagemaker-training-jobs/)"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "rapids",
   "language": "python",
   "name": "rapids"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  },
  "vscode": {
   "interpreter": {
    "hash": "f7a54d993f849a0f97fda357a1a3bac7e25a43aff77e618e8d69a4ad36661dba"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}