tfx-pipeline-lab-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Continuous training with TFX and Google Cloud AI Platform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Objectives\n",
    "\n",
    "1.  Use the TFX CLI to build a TFX pipeline.\n",
    "2.  Deploy a TFX pipeline version without tuning to a hosted AI Platform Pipelines instance.\n",
    "3.  Create and monitor a TFX pipeline run using the TFX CLI.\n",
    "4.  Deploy a new TFX pipeline version with tuning enabled to a hosted AI Platform Pipelines instance.\n",
    "5.  Create and monitor another TFX pipeline run directly in the KFP UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you use utilize the following tools and services to deploy and run a TFX pipeline on Google Cloud that automates the development and deployment of a TensorFlow 2.3 WideDeep Classifer to predict forest cover from cartographic data:\n",
    "\n",
    "* The [**TFX CLI**](https://www.tensorflow.org/tfx/guide/cli) utility to build and deploy a TFX pipeline.\n",
    "* A hosted [**AI Platform Pipeline instance (Kubeflow Pipelines)**](https://www.tensorflow.org/tfx/guide/kubeflow) for TFX pipeline orchestration.\n",
    "* [**Dataflow**](https://cloud.google.com/dataflow) jobs for scalable, distributed data processing for TFX components.\n",
    "* A [**AI Platform Training**](https://cloud.google.com/ai-platform/) job for model training and flock management for parallel tuning trials. \n",
    "* [**AI Platform Prediction**](https://cloud.google.com/ai-platform/) as a model server destination for blessed pipeline model versions.\n",
    "* [**CloudTuner**](https://www.tensorflow.org/tfx/guide/tuner#tuning_on_google_cloud_platform_gcp) and [**AI Platform Vizier**](https://cloud.google.com/ai-platform/optimizer/docs/overview) for advanced model hyperparameter tuning using the Vizier algorithm.\n",
    "\n",
    "You will then create and monitor pipeline runs using the TFX CLI as well as the KFP UI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Update lab environment PATH to include TFX CLI and skaffold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "# Set `PATH` to include the directory containing TFX CLI and skaffold.\n",
    "PATH=%env PATH\n",
    "%env PATH=/home/jupyter/.local/bin:/local/bin/:{PATH}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validate lab package version installation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -c \"import tfx; print('TFX version: {}'.format(tfx.__version__))\"\n",
    "!python -c \"import kfp; print('KFP version: {}'.format(kfp.__version__))\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: this lab was built and tested with the following package versions:\n",
    "\n",
    "`TFX version: 0.25.0`  \n",
    "`KFP version: 1.0.4`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Optional) If running the above command results in different package versions or you receive an import error, upgrade to the correct versions by running the cell below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --user tfx==0.25.0\n",
    "%pip install --upgrade --user kfp==1.0.4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: you may need to restart the kernel to pick up the correct package versions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validate creation of AI Platform Pipelines cluster\n",
    "\n",
    "Navigate to [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines/clusters) page in the Google Cloud Console.\n",
    "\n",
    "Note you may have already deployed an AI Pipelines instance during the Setup for the lab series. If so, you can proceed using that instance. If not:\n",
    "\n",
    "**1.  Create or select an existing Kubernetes cluster (GKE) and deploy AI Platform**. Make sure to select `\"Allow access to the following Cloud APIs https://www.googleapis.com/auth/cloud-platform\"` to allow for programmatic access to your pipeline by the Kubeflow SDK for the rest of the lab. Also, provide an `App instance name` such as \"tfx\" or \"mlops\". \n",
    "\n",
    "Validate the deployment of your AI Platform Pipelines instance in the console before proceeding."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Review: example TFX pipeline design pattern for Google Cloud\n",
    "The pipeline source code can be found in the `pipeline` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls -la"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `config.py` module configures the default values for the environment specific settings and the default values for the pipeline runtime parameters. \n",
    "The default values can be overwritten at compile time by providing the updated values in a set of environment variables. You will set custom environment variables later on this lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pipeline.py` module contains the TFX DSL defining the workflow implemented by the pipeline.\n",
    "\n",
    "The `preprocessing.py` module implements the data preprocessing logic  the `Transform` component.\n",
    "\n",
    "The `model.py` module implements the training, tuning, and model building logic for the `Trainer` and `Tuner` components.\n",
    "\n",
    "The `runner.py` module configures and executes `KubeflowDagRunner`. At compile time, the `KubeflowDagRunner.run()` method converts the TFX DSL into the pipeline package in the [argo](https://argoproj.github.io/argo/) format for execution on your hosted AI Platform Pipelines instance.\n",
    "\n",
    "The `features.py` module contains feature definitions common across `preprocessing.py` and `model.py`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: build your pipeline with the TFX CLI\n",
    "\n",
    "You will use TFX CLI to compile and deploy the pipeline. As explained in the previous section, the environment specific settings can be provided through a set of environment variables and embedded into the pipeline package at compile time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure your environment resource settings\n",
    "\n",
    "Update  the below constants  with the settings reflecting your lab environment. \n",
    "\n",
    "- `GCP_REGION` - the compute region for AI Platform Training, Vizier, and Prediction.\n",
    "- `ARTIFACT_STORE` - An existing GCS bucket. You can use any bucket or use the GCS bucket created during installation of AI Platform Pipelines. The default bucket name will contain the `kubeflowpipelines-` prefix. When specifying the bucket, do not use the trailing slash (/) at the end of the bucket name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use the following command to identify the GCS bucket for metadata and pipeline storage.\n",
    "!gsutil ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* `CUSTOM_SERVICE_ACCOUNT` - In the gcp console Click on the Navigation Menu. Navigate to `IAM & Admin`, then to `Service Accounts` and use the service account starting with prefix - `'tfx-tuner-caip-service-account'`. This enables CloudTuner and the Google Cloud AI Platform extensions Tuner component to work together and allows for distributed and parallel tuning backed by AI Platform Vizier's hyperparameter search algorithm. Please see the lab setup `README` for setup instructions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `ENDPOINT` - set the `ENDPOINT` constant to the endpoint to your AI Platform Pipelines instance. The endpoint to the AI Platform Pipelines instance can be found on the [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines/clusters) page in the Google Cloud Console. Open the *SETTINGS* for your instance and use the value of the `host` variable in the *Connect to this Kubeflow Pipelines instance from a Python client via Kubeflow Pipelines SKD* section of the *SETTINGS* window. The format is `'...pipelines.googleusercontent.com'`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pwd\n",
    "!ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#TODO: Set your environment resource settings here for GCP_REGION, ARTIFACT_STORE_URI, ENDPOINT, and CUSTOM_SERVICE_ACCOUNT.\n",
    "GCP_REGION = 'us-central1-a'\n",
    "ARTIFACT_STORE_URI = 'gs://mlops-workshop-artifact-store'\n",
    "ENDPOINT = 'https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com'\n",
    "CUSTOM_SERVICE_ACCOUNT = 'tfx-tuner-caip-service-account@qwiklabs-gcp-00-a07d99a55b52.iam.gserviceaccount.com'\n",
    "\n",
    "PROJECT_ID = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_ID[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set your resource settings as environment variables. These override the default values in pipeline/config.py.\n",
    "%env GCP_REGION={GCP_REGION}\n",
    "%env ARTIFACT_STORE_URI={ARTIFACT_STORE_URI}\n",
    "%env CUSTOM_SERVICE_ACCOUNT={CUSTOM_SERVICE_ACCOUNT}\n",
    "%env PROJECT_ID={PROJECT_ID}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set the compile time settings to first create a pipeline version without hyperparameter tuning\n",
    "\n",
    "Default pipeline runtime environment values are configured in the pipeline folder `config.py`. You will set their values directly below:\n",
    "\n",
    "* `PIPELINE_NAME` - the pipeline's globally unique name. For each pipeline update, each pipeline version uploaded to KFP will be reflected on the `Pipelines` tab in the `Pipeline name > Version name` dropdown in the format `PIPELINE_NAME_datetime.now()`.\n",
    "\n",
    "* `MODEL_NAME` - the pipeline's unique model output name for AI Platform Prediction. For multiple pipeline runs, each pushed blessed model will create a new version with the format `'v{}'.format(int(time.time()))`.\n",
    "\n",
    "* `DATA_ROOT_URI` - the URI for the raw lab dataset `gs://cloud-training/OCBL203/workshop-datasets`.\n",
    "\n",
    "* `CUSTOM_TFX_IMAGE` - the image name of your pipeline container build by skaffold and published by `Cloud Build` to `Cloud Container Registry` in the format `'gcr.io/{}/{}'.format(PROJECT_ID, PIPELINE_NAME)`.\n",
    "\n",
    "* `RUNTIME_VERSION` - the TensorFlow runtime version. This lab was built and tested using TensorFlow `2.3`.\n",
    "\n",
    "* `PYTHON_VERSION` - the Python runtime version. This lab was built and tested using Python `3.7`.\n",
    "\n",
    "* `USE_KFP_SA` - The pipeline can run using a security context of the GKE default node pool's service account or the service account defined in the `user-gcp-sa` secret of the Kubernetes namespace hosting Kubeflow Pipelines. If you want to use the `user-gcp-sa` service account you change the value of `USE_KFP_SA` to `True`. Note that the default AI Platform Pipelines configuration does not define the `user-gcp-sa` secret.\n",
    "\n",
    "* `ENABLE_TUNING` - boolean value indicating whether to add the `Tuner` component to the pipeline or use hyperparameter defaults. See the `model.py` and `pipeline.py` files for details on how this changes the pipeline topology across pipeline versions. You will create pipeline versions without and with tuning enabled in the subsequent lab exercises for comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!ls /home/jupyter/.local/lib/python3.7/site-packages/tfx/tools/cli/container_builder/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_NAME = 'tfx_covertype_continuous_training'\n",
    "MODEL_NAME = 'tfx_covertype_classifier'\n",
    "DATA_ROOT_URI = 'gs://cloud-training/OCBL203/workshop-datasets'\n",
    "CUSTOM_TFX_IMAGE = 'gcr.io/{}/{}'.format(PROJECT_ID, PIPELINE_NAME)\n",
    "RUNTIME_VERSION = '2.3'\n",
    "PYTHON_VERSION = '3.7'\n",
    "USE_KFP_SA=False\n",
    "ENABLE_TUNING=False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env PIPELINE_NAME={PIPELINE_NAME}\n",
    "%env MODEL_NAME={MODEL_NAME}\n",
    "%env DATA_ROOT_URI={DATA_ROOT_URI}\n",
    "%env KUBEFLOW_TFX_IMAGE={CUSTOM_TFX_IMAGE}\n",
    "%env RUNTIME_VERSION={RUNTIME_VERSION}\n",
    "%env PYTHON_VERIONS={PYTHON_VERSION}\n",
    "%env USE_KFP_SA={USE_KFP_SA}\n",
    "%env ENABLE_TUNING={ENABLE_TUNING}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile your pipeline code\n",
    "\n",
    "You can build and upload the pipeline to the AI Platform Pipelines instance in one step, using the `tfx pipeline create` command. The `tfx pipeline create` goes through the following steps:\n",
    "- (Optional) Builds the custom image to that provides a runtime environment for TFX components or uses the latest image of the installed TFX version \n",
    "- Compiles the pipeline code into a pipeline package \n",
    "- Uploads the pipeline package via the `ENDPOINT` to the hosted AI Platform instance.\n",
    "\n",
    "As you debug the pipeline DSL, you may prefer to first use the `tfx pipeline compile` command, which only executes the compilation step. After the DSL compiles successfully you can use `tfx pipeline create` to go through all steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx pipeline compile --engine kubeflow --pipeline_path runner.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: you should see a `{PIPELINE_NAME}.tar.gz` file appear in your current pipeline directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise: deploy your pipeline container to AI Platform Pipelines with TFX CLI\n",
    "\n",
    "After the pipeline code compiles without any errors you can use the `tfx pipeline create` command to perform the full build and deploy the pipeline. You will deploy your compiled pipeline container hosted on Google Container Registry e.g. `gcr.io/[PROJECT_ID]/tfx_covertype_continuous_training` to run on AI Platform Pipelines with the TFX CLI."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Skaffold] c4e66be694ce: Layer already exists\n",
      "[Skaffold] 47cc65c6dd57: Layer already exists\n",
      "[Skaffold] 13b8ba6d955a: Pushed\n",
      "[Skaffold] 264aaaede9ad: Pushed\n",
      "[Skaffold] latest: digest: sha256:77e9de229adebb0b1f87e93a55d57ff6b69869714754cdce5272bf64ac0bd22c size: 3270\n",
      "[Skaffold] Build [gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training] succeeded\n",
      "[Skaffold] \n",
      "New container image is built. Target image is available in the build spec file.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "\u001b[0mPipeline compiled successfully.\n",
      "Pipeline package path: /home/jupyter/mlops-on-gcp/workshops/tfx-caip-tf23/lab-02-tfx-pipeline/labs/pipeline/tfx_covertype_continuous_training.tar.gz\n",
      "{'created_at': datetime.datetime(2023, 6, 16, 7, 49, 47, tzinfo=tzlocal()),\n",
      " 'default_version': {'code_source_url': None,\n",
      "                     'created_at': datetime.datetime(2023, 6, 16, 7, 49, 47, tzinfo=tzlocal()),\n",
      "                     'description': None,\n",
      "                     'id': '104e7b17-6a24-4753-802d-d984c65f25c7',\n",
      "                     'name': 'tfx_covertype_continuous_training',\n",
      "                     'package_url': None,\n",
      "                     'parameters': [{'name': 'pipeline-root',\n",
      "                                     'value': 'gs://mlops-workshop-artifact-store/tfx_covertype_continuous_training/{{workflow.uid}}'},\n",
      "                                    {'name': 'data-root-uri',\n",
      "                                     'value': 'gs://cloud-training/OCBL203/workshop-datasets'},\n",
      "                                    {'name': 'eval-steps', 'value': '500'},\n",
      "                                    {'name': 'train-steps', 'value': '5000'}],\n",
      "                     'resource_references': [{'key': {'id': '104e7b17-6a24-4753-802d-d984c65f25c7',\n",
      "                                                      'type': 'PIPELINE'},\n",
      "                                              'name': None,\n",
      "                                              'relationship': 'OWNER'}]},\n",
      " 'description': None,\n",
      " 'error': None,\n",
      " 'id': '104e7b17-6a24-4753-802d-d984c65f25c7',\n",
      " 'name': 'tfx_covertype_continuous_training',\n",
      " 'parameters': [{'name': 'pipeline-root',\n",
      "                 'value': 'gs://mlops-workshop-artifact-store/tfx_covertype_continuous_training/{{workflow.uid}}'},\n",
      "                {'name': 'data-root-uri',\n",
      "                 'value': 'gs://cloud-training/OCBL203/workshop-datasets'},\n",
      "                {'name': 'eval-steps', 'value': '500'},\n",
      "                {'name': 'train-steps', 'value': '5000'}],\n",
      " 'resource_references': None,\n",
      " 'url': None}\n",
      "Please access the pipeline detail page at https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com/#/pipelines/details/104e7b17-6a24-4753-802d-d984c65f25c7\n",
      "Pipeline \"tfx_covertype_continuous_training\" created successfully.\n"
     ]
    }
   ],
   "source": [
    "# TODO: Your code here to use the TFX CLI to deploy your pipeline image to AI Platform Pipelines.\n",
    "!tfx pipeline create  \\\n",
    "--pipeline_path=runner.py \\\n",
    "--endpoint={ENDPOINT} \\\n",
    "--build_target_image={CUSTOM_TFX_IMAGE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hint**: review the [TFX CLI documentation](https://www.tensorflow.org/tfx/guide/cli#create) on the \"pipeline group\" to create your pipeline. You will need to specify the `--pipeline_path` to point at the pipeline DSL and runner defined locally in `runner.py`, `--endpoint`, and `--build_target_image` arguments using the environment variables specified above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: you should see a `build.yaml` file in your pipeline folder created by skaffold. The TFX CLI compile triggers a custom container to be built with skaffold using the instructions in the `Dockerfile`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to redeploy the pipeline you can first delete the previous version using `tfx pipeline delete` or you can update the pipeline in-place using `tfx pipeline update`.\n",
    "\n",
    "To delete the pipeline:\n",
    "\n",
    "`tfx pipeline delete --pipeline_name {PIPELINE_NAME} --endpoint {ENDPOINT}`\n",
    "\n",
    "To update the pipeline:\n",
    "\n",
    "`tfx pipeline update --pipeline_path runner.py --endpoint {ENDPOINT}`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create and monitor a pipeline run with the TFX CLI\n",
    "\n",
    "After the pipeline has been deployed, you can trigger and monitor pipeline runs using TFX CLI.\n",
    "\n",
    "*Hint*: review the [TFX CLI documentation](https://www.tensorflow.org/tfx/guide/cli#run_group) on the \"run group\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLI\n",
      "Creating a run for pipeline: tfx_covertype_continuous_training\n",
      "Detected Kubeflow.\n",
      "Use --engine flag if you intend to use a different orchestrator.\n",
      "Run created for pipeline: tfx_covertype_continuous_training\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n",
      "| pipeline_name                     | run_id                               | status   | created_at                | link                                                                                                                         |\n",
      "+===================================+======================================+==========+===========================+==============================================================================================================================+\n",
      "| tfx_covertype_continuous_training | 7e3427f2-078a-460a-bf8f-b0b3ee894dfa |          | 2023-06-16T07:52:19+00:00 | https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com/#/runs/details/7e3427f2-078a-460a-bf8f-b0b3ee894dfa |\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "# TODO: your code here to trigger a pipeline run with the TFX CLI\n",
    "!tfx run create --pipeline_name={PIPELINE_NAME} --endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the status of existing pipeline runs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLI\n",
      "Listing all runs of pipeline: tfx_covertype_continuous_training\n",
      "Detected Kubeflow.\n",
      "Use --engine flag if you intend to use a different orchestrator.\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n",
      "| pipeline_name                     | run_id                               | status   | created_at                | link                                                                                                                         |\n",
      "+===================================+======================================+==========+===========================+==============================================================================================================================+\n",
      "| tfx_covertype_continuous_training | 7e3427f2-078a-460a-bf8f-b0b3ee894dfa | Running  | 2023-06-16T07:52:19+00:00 | https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com/#/runs/details/7e3427f2-078a-460a-bf8f-b0b3ee894dfa |\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "!tfx run list --pipeline_name {PIPELINE_NAME} --endpoint {ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To retrieve the status of a given run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLI\n",
      "Retrieving run status.\n",
      "Detected Kubeflow.\n",
      "Use --engine flag if you intend to use a different orchestrator.\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n",
      "| pipeline_name                     | run_id                               | status   | created_at                | link                                                                                                                         |\n",
      "+===================================+======================================+==========+===========================+==============================================================================================================================+\n",
      "| tfx_covertype_continuous_training | 7e3427f2-078a-460a-bf8f-b0b3ee894dfa | Failed   | 2023-06-16T07:52:19+00:00 | https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com/#/runs/details/7e3427f2-078a-460a-bf8f-b0b3ee894dfa |\n",
      "+-----------------------------------+--------------------------------------+----------+---------------------------+------------------------------------------------------------------------------------------------------------------------------+\n"
     ]
    }
   ],
   "source": [
    "RUN_ID='7e3427f2-078a-460a-bf8f-b0b3ee894dfa'\n",
    "\n",
    "!tfx run status --pipeline_name {PIPELINE_NAME} --run_id {RUN_ID} --endpoint {ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Important \n",
    "\n",
    "A full pipeline run without tuning enabled will take about 40 minutes to complete. You can view the run's progress using the TFX CLI commands above or in the"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Optional Exercise: deploy a pipeline version with tuning enabled"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Incorporating automatic model hyperparameter tuning into a continuous training TFX pipeline workflow enables faster experimentation, development, and deployment of a top performing model.\n",
    "\n",
    "The previous pipeline version read from hyperparameter default values in the search space defined in `_get_hyperparameters()` in `model.py` and used these values to build a TensorFlow WideDeep Classifier model.\n",
    "\n",
    "Let's now deploy a new pipeline version with the `Tuner` component added to the pipeline that calls out to the AI Platform Vizier service for distributed and parallelized hyperparameter tuning. The `Tuner` component `\"best_hyperparameters\"` artifact will be passed directly to your `Trainer` component to deploy the top performing model. Review `pipeline.py` to see how this environment variable changes the pipeline topology. Also, review the tuning function in `model.py` for configuring `CloudTuner`.\n",
    "\n",
    "Note that you might not want to tune the hyperparameters every time you retrain your model due to the computational cost. Once you have used `Tuner` determine a good set of hyperparameters, you can remove `Tuner` from your pipeline and use model hyperparameters defined in your model code or use a `ImporterNode` to import the `Tuner` `\"best_hyperparameters\"`artifact from a previous `Tuner` run to your model `Trainer`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENABLE_TUNING=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: ENABLE_TUNING=True\n"
     ]
    }
   ],
   "source": [
    "%env ENABLE_TUNING={ENABLE_TUNING}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compile your pipeline code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLI\n",
      "Compiling pipeline\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "\u001b[0mPipeline compiled successfully.\n",
      "Pipeline package path: /home/jupyter/mlops-on-gcp/workshops/tfx-caip-tf23/lab-02-tfx-pipeline/labs/pipeline/tfx_covertype_continuous_training.tar.gz\n"
     ]
    }
   ],
   "source": [
    "!tfx pipeline compile --engine kubeflow --pipeline_path runner.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy your pipeline container to AI Platform Pipelines with the TFX CLI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CLI\n",
      "Updating pipeline\n",
      "Detected Kubeflow.\n",
      "Use --engine flag if you intend to use a different orchestrator.\n",
      "Reading build spec from build.yaml\n",
      "[Skaffold] Generating tags...\n",
      "[Skaffold]  - gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training -> gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training:latest\n",
      "[Skaffold] Checking cache...\n",
      "[Skaffold]  - gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training: Not found. Building\n",
      "[Skaffold] Starting build...\n",
      "[Skaffold] Building [gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training]...\n",
      "[Skaffold] #1 [internal] load build definition from Dockerfile\n",
      "[Skaffold] #1 transferring dockerfile: 31B done\n",
      "[Skaffold] #1 DONE 0.2s\n",
      "[Skaffold] \n",
      "[Skaffold] #2 [internal] load .dockerignore\n",
      "[Skaffold] #2 transferring context: 2B done\n",
      "[Skaffold] #2 DONE 0.2s\n",
      "[Skaffold] \n",
      "[Skaffold] #3 [internal] load metadata for docker.io/tensorflow/tfx:0.25.0\n",
      "[Skaffold] #3 DONE 0.3s\n",
      "[Skaffold] \n",
      "[Skaffold] #4 [1/3] FROM docker.io/tensorflow/tfx:0.25.0@sha256:0700c27c6492b8b2998e7d543ca13088db8d40ef26bd5c6eec58245ff8cdec35\n",
      "[Skaffold] #4 DONE 0.0s\n",
      "[Skaffold] \n",
      "[Skaffold] #5 [internal] load build context\n",
      "[Skaffold] #5 transferring context: 17.97kB done\n",
      "[Skaffold] #5 DONE 0.0s\n",
      "[Skaffold] \n",
      "[Skaffold] #6 [2/3] WORKDIR ./pipeline\n",
      "[Skaffold] #6 CACHED\n",
      "[Skaffold] \n",
      "[Skaffold] #7 [3/3] COPY ./ ./\n",
      "[Skaffold] #7 DONE 1.4s\n",
      "[Skaffold] \n",
      "[Skaffold] #8 exporting to image\n",
      "[Skaffold] #8 exporting layers\n",
      "[Skaffold] #8 exporting layers 1.2s done\n",
      "[Skaffold] #8 writing image sha256:ba6401554192f03486e6ef0ec822d4db5ef42c96d8e31f785eabb5b8b385fca4 0.0s done\n",
      "[Skaffold] #8 naming to gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training:latest 0.0s done\n",
      "[Skaffold] #8 DONE 1.3s\n",
      "[Skaffold] The push refers to repository [gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training]\n",
      "[Skaffold] 6580429e4a3d: Preparing\n",
      "[Skaffold] 13b8ba6d955a: Preparing\n",
      "[Skaffold] 5dadc0a09248: Preparing\n",
      "[Skaffold] 8fb12d3bda49: Preparing\n",
      "[Skaffold] 2471eac28ba8: Preparing\n",
      "[Skaffold] 674ba689ae71: Preparing\n",
      "[Skaffold] 4058ae03fa32: Preparing\n",
      "[Skaffold] e3437c61d457: Preparing\n",
      "[Skaffold] 84ff92691f90: Preparing\n",
      "[Skaffold] 54b00d861a7a: Preparing\n",
      "[Skaffold] c547358928ab: Preparing\n",
      "[Skaffold] 84ff92691f90: Preparing\n",
      "[Skaffold] c4e66be694ce: Preparing\n",
      "[Skaffold] 47cc65c6dd57: Preparing\n",
      "[Skaffold] 674ba689ae71: Waiting\n",
      "[Skaffold] 4058ae03fa32: Waiting\n",
      "[Skaffold] e3437c61d457: Waiting\n",
      "[Skaffold] 84ff92691f90: Waiting\n",
      "[Skaffold] 54b00d861a7a: Waiting\n",
      "[Skaffold] c547358928ab: Waiting\n",
      "[Skaffold] c4e66be694ce: Waiting\n",
      "[Skaffold] 47cc65c6dd57: Waiting\n",
      "[Skaffold] 13b8ba6d955a: Layer already exists\n",
      "[Skaffold] 2471eac28ba8: Layer already exists\n",
      "[Skaffold] 5dadc0a09248: Layer already exists\n",
      "[Skaffold] 8fb12d3bda49: Layer already exists\n",
      "[Skaffold] 4058ae03fa32: Layer already exists\n",
      "[Skaffold] 674ba689ae71: Layer already exists\n",
      "[Skaffold] e3437c61d457: Layer already exists\n",
      "[Skaffold] 84ff92691f90: Layer already exists\n",
      "[Skaffold] 54b00d861a7a: Layer already exists\n",
      "[Skaffold] c547358928ab: Layer already exists\n",
      "[Skaffold] c4e66be694ce: Layer already exists\n",
      "[Skaffold] 47cc65c6dd57: Layer already exists\n",
      "[Skaffold] 6580429e4a3d: Pushed\n",
      "[Skaffold] latest: digest: sha256:83347fab9cff285c3a0300e2adc3f110019df57960b3ebe302d108919c932516 size: 3270\n",
      "[Skaffold] Build [gcr.io/qwiklabs-gcp-00-a07d99a55b52/tfx_covertype_continuous_training] succeeded\n",
      "[Skaffold] \n",
      "New container image is built. Target image is available in the build spec file.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:RuntimeParameter is only supported on Cloud-based DAG runner currently.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "WARNING:absl:`instance_name` is deprecated, please set node id directly using`with_id()` or `.id` setter.\n",
      "\u001b[0mPipeline compiled successfully.\n",
      "Pipeline package path: /home/jupyter/mlops-on-gcp/workshops/tfx-caip-tf23/lab-02-tfx-pipeline/labs/pipeline/tfx_covertype_continuous_training.tar.gz\n",
      "{'code_source_url': None,\n",
      " 'created_at': datetime.datetime(2023, 6, 16, 8, 15, 52, tzinfo=tzlocal()),\n",
      " 'description': None,\n",
      " 'id': '9c4cdf01-b6e6-414e-afb3-b33223d5b90d',\n",
      " 'name': 'tfx_covertype_continuous_training_20230616081552',\n",
      " 'package_url': None,\n",
      " 'parameters': [{'name': 'pipeline-root',\n",
      "                 'value': 'gs://mlops-workshop-artifact-store/tfx_covertype_continuous_training/{{workflow.uid}}'},\n",
      "                {'name': 'data-root-uri',\n",
      "                 'value': 'gs://cloud-training/OCBL203/workshop-datasets'},\n",
      "                {'name': 'eval-steps', 'value': '500'},\n",
      "                {'name': 'train-steps', 'value': '5000'}],\n",
      " 'resource_references': [{'key': {'id': '104e7b17-6a24-4753-802d-d984c65f25c7',\n",
      "                                  'type': 'PIPELINE'},\n",
      "                          'name': None,\n",
      "                          'relationship': 'OWNER'}]}\n",
      "Please access the pipeline detail page at https://53f139f15d67eaa6-dot-us-central1.pipelines.googleusercontent.com/#/pipelines/details/104e7b17-6a24-4753-802d-d984c65f25c7\n",
      "Pipeline \"tfx_covertype_continuous_training\" updated successfully.\n"
     ]
    }
   ],
   "source": [
    "#TODO: your code to update your pipeline \n",
    "!tfx pipeline update --pipeline_path runner.py --endpoint {ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigger a pipeline run from the Kubeflow Pipelines UI\n",
    "\n",
    "On the [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines/clusters) page, click `OPEN PIPELINES DASHBOARD`. A new browser tab will open. Select the `Pipelines` tab to the left where you see the `PIPELINE_NAME` pipeline you deployed previously. You should see 2 pipeline versions. \n",
    "\n",
    "Click on the most recent pipeline version with tuning enabled which will open up a window with a graphical display of your TFX pipeline directed graph. \n",
    "\n",
    "Next, click the `Create a run` button. Verify the `Pipeline name` and `Pipeline version` are pre-populated and optionally provide a `Run name` and `Experiment` to logically group the run metadata under before hitting `Start` to trigger the pipeline run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Important\n",
    "\n",
    "A full pipeline run with tuning enabled will take about 50 minutes and can be executed in parallel while the previous pipeline run without tuning continues running. \n",
    "\n",
    "Take the time to review the pipeline metadata artifacts created in the GCS artifact repository for each component including data splits, your Tensorflow SavedModel, model evaluation results, etc. as the pipeline executes. In the GCP console, you can also view the Dataflow jobs for pipeline data processing as well as the AI Platform Training jobs for model training and tuning.\n",
    "\n",
    "When your pipelines runs are complete, review your model versions on Cloud AI Platform Prediction and model evaluation metrics. Did your model performance improve with hyperparameter tuning?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next Steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab, you learned how to build and deploy a TFX pipeline with the TFX CLI and then update, build and deploy a new pipeline with automatic hyperparameter tuning. You practiced triggered continuous pipeline runs using the TFX CLI as well as the Kubeflow Pipelines UI.\n",
    "\n",
    "\n",
    "In the next lab, you will construct a Cloud Build CI/CD workflow that further automates the building and deployment of the TensorFlow WideDeep Classifer pipeline code introduced in this lab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-root-py",
   "name": "tf2-cpu.2-3.m87",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-cpu.2-3:m87"
  },
  "kernelspec": {
   "display_name": "Python [conda env:root] *",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}