optuna/notebooks/azure-optuna/run_optuna.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training RAPIDS + OPtuna experiment on Azure\n",
    "\n",
    "## Prerequisites\n",
    "- Create an Azure ML Workspace and setup environmnet on local computer following the steps in [Azure README.md](https://github.com/rapidsai/cloud-ml-examples/blob/main/azure/README.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# verify installation and check Azure ML SDK version\n",
    "import azureml.core\n",
    "\n",
    "print('SDK version:', azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.workspace import Workspace\n",
    "\n",
    "ws = Workspace.from_config()\n",
    "print('Workspace name: ' + ws.name, \n",
    "      'Azure region: ' + ws.location, \n",
    "      'Subscription id: ' + ws.subscription_id, \n",
    "      'Resource group: ' + ws.resource_group, sep = '\\n')\n",
    "\n",
    "datastore = ws.get_default_datastore()\n",
    "print(\"Default datastore's name: {}\".format(datastore.name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create AML compute\n",
    "\n",
    "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for training your model. In this notebook, we will use Azure ML managed compute ([AmlCompute](https://docs.microsoft.com/azure/machine-learning/service/how-to-set-up-training-targets#amlcompute)) for our remote training using a dynamically scalable pool of compute resources.\n",
    "\n",
    "This notebook will use 10 nodes for hyperparameter optimization, you can modify `max_node` based on available quota in the desired region. Similar to other Azure services, there are limits on certain resources (e.g. AmlCompute) associated with the Azure Machine Learning service. [This article](https://docs.microsoft.com/en-us/azure/machine-learning/service/how-to-manage-quotas) includes details on the default limits and how to request more quota.\n",
    "\n",
    "`vm_size` describes the virtual machine type and size that will be used in the cluster. RAPIDS requires NVIDIA Pascal or newer architecture, you will need to specify compute targets from one of `NC_v2`, `NC_v3`, `ND` or `ND_v2` [GPU virtual machines in Azure](https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu); these are VMs that are provisioned with P40 and V100 GPUs. Let's create an `AmlCompute` cluster of `Standard_NC6s_v3` GPU VMs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException\n",
    "\n",
    "# choose a name for your cluster\n",
    "gpu_cluster_name = 'gpu-cluster'\n",
    "\n",
    "if gpu_cluster_name in ws.compute_targets:\n",
    "    gpu_cluster = ws.compute_targets[gpu_cluster_name]\n",
    "    if gpu_cluster and type(gpu_cluster) is AmlCompute:\n",
    "        print('Found compute target. Will use {0} '.format(gpu_cluster_name))\n",
    "else:\n",
    "    print('creating new cluster')\n",
    "    # m_size parameter below could be modified to one of the RAPIDS-supported VM types\n",
    "    provisioning_config = AmlCompute.provisioning_configuration(vm_size = 'Standard_NC6s_v3', max_nodes = 5, idle_seconds_before_scaledown = 300)\n",
    "    # Use VM types with more than one GPU for multi-GPU option, e.g. Standard_NC12s_v3\n",
    "    \n",
    "    # create the cluster\n",
    "    gpu_cluster = ComputeTarget.create(ws, gpu_cluster_name, provisioning_config)\n",
    "    \n",
    "    # can poll for a minimum number of nodes and for a specific timeout \n",
    "    # if no min node count is provided it uses the scale settings for the cluster\n",
    "    gpu_cluster.wait_for_completion(show_output=True, min_node_count=None, timeout_in_minutes=20)\n",
    "    \n",
    "# use get_status() to get a detailed status for the current cluster \n",
    "print(gpu_cluster.get_status().serialize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create experiment\n",
    "\n",
    "Create an [Experiment](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#experiment) to track all the runs in your workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Experiment\n",
    "\n",
    "experiment_name = 'optuna_rapids'\n",
    "experiment = Experiment(ws, name=experiment_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Environment class](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.environment.environment?view=azure-ml-py) allows you to build a Docker image and customize the system that you will use for training. We will build a container image using a RAPIDS container as base image and install necessary packages. This build is necessary only the first time and will take about 15 minutes. The image will be added to your Azure Container Registry and the environment will be cached after the first run, as long as the environment definition remains the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Environment\r\n",
    "\r\n",
    "# create the environment\r\n",
    "rapids_env = Environment('rapids_env')\r\n",
    "\r\n",
    "# create the environment inside a Docker container\r\n",
    "rapids_env.docker.enabled = True\r\n",
    "\r\n",
    "# specify docker steps as a string. Alternatively, load the string from a file\r\n",
    "dockerfile = \"\"\"\r\n",
    "FROM rapidsai/rapidsai:21.06-cuda10.2-runtime-ubuntu18.04-py3.7\r\n",
    "RUN apt-get update && \\\r\n",
    "apt-get install -y fuse && \\\r\n",
    "apt-get install libssl1.0.0 libssl-dev && \\\r\n",
    "source activate rapids && \\\r\n",
    "pip install azureml-sdk==1.13.0 && \\\r\n",
    "pip install azureml-widgets && \\\r\n",
    "pip install optuna && \\\r\n",
    "pip install dask_optuna && \\\r\n",
    "pip install fusepy\r\n",
    "\"\"\"\r\n",
    "\r\n",
    "# set base image to None since the image is defined by dockerfile\r\n",
    "rapids_env.docker.enabled = True\r\n",
    "rapids_env.docker.base_image = None\r\n",
    "rapids_env.docker.base_dockerfile = dockerfile\r\n",
    "\r\n",
    "# use rapids environment in the container\r\n",
    "rapids_env.python.user_managed_dependencies = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acquiring Data\n",
    "\n",
    "1. Download data from Kaggle - [BNP Paribas Carid Calims Management](https://www.kaggle.com/c/bnp-paribas-cardif-claims-management/data)\n",
    "2. Place only the unzipped `train.csv` file in `data_dir`(by default this is set to `'data/'`)\n",
    "3. The following cell will upload it to your default workspace datastore under `path_on_datastore`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "data_dir = \"data/\"\n",
    "path_on_datastore = 'bnp_upload'\n",
    "\n",
    "datastore.upload(src_dir=data_dir, target_path=path_on_datastore, overwrite=False, show_progress=True)\n",
    "\n",
    "ds_data = datastore.path(path_on_datastore)\n",
    "dataset = Dataset.File.from_files(ds_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_params = ['--data_dir', dataset.as_named_input('bnp_input').as_mount(),\n",
    "]\n",
    "from azureml.core import ScriptRunConfig\n",
    "\n",
    "project_folder = \"project_folder/\"\n",
    "src = ScriptRunConfig(source_directory=project_folder,\n",
    "                      script='train_optuna.py',\n",
    "                      arguments=script_params,\n",
    "                      compute_target=\"gpu-cluster\",\n",
    "                      environment=rapids_env)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run = experiment.submit(config=src)\n",
    "from azureml.widgets import RunDetails\n",
    "RunDetails(run).show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "run.wait_for_completion(show_output=False)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "890835e6f88d29526c142dfe60f39e556c40454df21a638b17c054b5521d5de3"
  },
  "kernelspec": {
   "display_name": "Python 3.8.8 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": ""
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2
}