-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding multiple outputs into sdk with sample (#1667)
* Added support for mulitple outputs * Added test for multiple output * Adding sample for multiple outputs * func_signature now shorter form * Added parameters tag * Fixed func_signature mistake
- Loading branch information
1 parent
8866747
commit f18b7fd
Showing
3 changed files
with
284 additions
and
11 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,210 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Multiple outputs example\n", | ||
| "This notebook is a simple example of how to make a component with multiple outputs using the Pipelines SDK.\n", | ||
| "\n", | ||
| "## Before running notebook:\n", | ||
| "\n", | ||
| "### Setup notebook server\n", | ||
| "This pipeline requires you to [setup a notebook server](https://www.kubeflow.org/docs/notebooks/setup/) in the Kubeflow UI. After you are setup, *upload the notebook in the Kubeflow UI* and then run it in the notebook server.\n", | ||
| "\n", | ||
| "### Create a GCS bucket\n", | ||
| "This pipeline requires a GCS bucket. If you haven't already, [create a GCS bucket](https://cloud.google.com/storage/docs/creating-buckets) to run the notebook. Make sure to create the storage bucket in the same project that you are running Kubeflow on to have the proper permissions by default. You can also create a GCS bucket by running `gsutil mb -p <project_name> gs://<bucket_name>`.\n", | ||
| "\n", | ||
| "### Upload the notebook in the Kubeflow UI\n", | ||
| "In order to run this pipeline, make sure to upload the notebook to your notebook server in the Kubeflow UI. You can clone this repo in the Jupyter notebook server by connecting to the notebook server and then selecting New > Terminal. In the terminal type `git clone https://github.com/kubeflow/pipelines.git`.\n", | ||
| "\n", | ||
| "### Install Kubeflow pipelines\n", | ||
| "Install the `kfp` package if you haven't already." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "scrolled": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "!pip install kfp --upgrade" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Setup project info and imports" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "tags": [ | ||
| "parameters" | ||
| ] | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "GCS_BUCKET = 'gs://[BUCKET-NAME]' # GCS bucket name\n", | ||
| "PROJECT_NAME = '[PROJECT-NAME]' # GCP project name\n", | ||
| "\n", | ||
| "STAGING_GCS_PATH = GCS_BUCKET + '/multiple-output-sample'\n", | ||
| "TARGET_IMAGE = 'gcr.io/%s/multi-output:latest' % PROJECT_NAME\n", | ||
| "\n", | ||
| "BASE_IMAGE = 'tensorflow/tensorflow:1.11.0-py3'\n", | ||
| "EXPERIMENT_NAME = 'Multiple Outputs Sample'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import kfp \n", | ||
| "import kfp.dsl as dsl\n", | ||
| "from kfp import compiler\n", | ||
| "from typing import NamedTuple" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Create component\n", | ||
| "In order to create a component with multiple outputs, use `NamedTuple` with the same syntax as below." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "# Returns a*b and a+b\n", | ||
| "@dsl.python_component(\n", | ||
| " name='product_sum',\n", | ||
| " description='Calculates the product and the sum',\n", | ||
| " base_image=BASE_IMAGE\n", | ||
| ")\n", | ||
| "def product_sum(a: float, b: float) -> NamedTuple(\n", | ||
| " 'output', [('product', float), ('sum', float)]):\n", | ||
| " '''Returns the product and sum of two numbers'''\n", | ||
| " from collections import namedtuple\n", | ||
| " \n", | ||
| " product_sum_output = namedtuple('output', ['product', 'sum'])\n", | ||
| " return product_sum_output(a*b, a+b)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "scrolled": true | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "product_sum_op = compiler.build_python_component(\n", | ||
| " component_func=product_sum,\n", | ||
| " staging_gcs_path=STAGING_GCS_PATH,\n", | ||
| " base_image=BASE_IMAGE,\n", | ||
| " target_image=TARGET_IMAGE)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Create and run pipeline\n", | ||
| "### Create pipeline\n", | ||
| "The pipeline parameters are specified in the `pipeline` function signature." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "@dsl.pipeline(\n", | ||
| " name='Multiple Outputs Pipeline',\n", | ||
| " description='Sample pipeline to showcase multiple outputs'\n", | ||
| ")\n", | ||
| "def pipeline(a=2.0, b=2.5, c=3.0):\n", | ||
| " prod_sum_task = product_sum_op(a, b)\n", | ||
| " prod_sum_task2 = product_sum_op(b, c)\n", | ||
| " prod_sum_task3 = product_sum_op(prod_sum_task.outputs['product'],\n", | ||
| " prod_sum_task2.outputs['sum'])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "pipeline_filename = 'multiple-outputs.pipelines.zip'\n", | ||
| "compiler.Compiler().compile(pipeline, pipeline_filename)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Run pipeline" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "client = kfp.Client()\n", | ||
| "experiment = client.create_experiment(EXPERIMENT_NAME)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "arguments = {\n", | ||
| " 'a': 2.0,\n", | ||
| " 'b': 2.5,\n", | ||
| " 'c': 3.0,\n", | ||
| "}\n", | ||
| "run_name = 'multiple output run'\n", | ||
| "run_result = client.run_pipeline(experiment.id, run_name, pipeline_filename,\n", | ||
| " params=arguments)" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.7.3" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters