Skip to content

Commit

Permalink
Action feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
@marckarp
marckarp committed Aug 17, 2022
1 parent df09eb2 commit 07cb67f
Showing 1 changed file with 38 additions and 34 deletions.
72 changes: 38 additions & 34 deletions sagemaker-triton/TensorFlow/Deploy-TensorFlow-Model-Using-NVIDIA-Triton.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,15 @@
"cells": [
{
"cell_type": "markdown",
"id": "3e47c49c",
"id": "e4df5204",
"metadata": {},
"source": [
"# Deploy a TensorFlow Model using NVIDIA Triton on SageMaker"
]
},
{
"cell_type": "markdown",
"id": "1d8fccde",
"id": "a263f99b",
"metadata": {},
"source": [
"Amazon SageMaker is a fully managed service for data science and machine learning workflows. It helps data scientists and developers to prepare, build, train, and deploy high-quality ML models quickly by bringing together a broad set of capabilities purpose-built for ML.\n",
Expand All @@ -34,7 +34,7 @@
},
{
"cell_type": "markdown",
"id": "ef62c715",
"id": "31e1faa2",
"metadata": {},
"source": [
"## Introduction to NVIDIA Triton Server\n",
Expand All @@ -55,7 +55,7 @@
},
{
"cell_type": "markdown",
"id": "9ed5bdfb",
"id": "26eeade9",
"metadata": {},
"source": [
"## Set up the environment\n",
Expand All @@ -67,7 +67,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "d0914d6e",
"id": "a3803a4b",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -83,7 +83,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "ab03eaef",
"id": "b0585270",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -97,7 +97,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "12c38fca",
"id": "0d337a66",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -130,7 +130,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "24e5f960",
"id": "c71f49a1",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -141,20 +141,22 @@
{
"cell_type": "code",
"execution_count": null,
"id": "2f272d83",
"id": "916daad1",
"metadata": {},
"outputs": [],
"source": [
"base = \"amazonaws.com.cn\" if region.startswith(\"cn-\") else \"amazonaws.com\"\n",
"triton_image_uri = \"{account_id}.dkr.ecr.{region}.{base}/sagemaker-tritonserver:21.08-py3\".format(\n",
" account_id=account_id_map[region], region=region, base=base\n",
"triton_image_uri = (\n",
" \"{account_id}.dkr.ecr.{region}.{base}/sagemaker-tritonserver:21.08-py3\".format(\n",
" account_id=account_id_map[region], region=region, base=base\n",
" )\n",
")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "1f5274ef",
"id": "cddaef96",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -163,7 +165,7 @@
},
{
"cell_type": "markdown",
"id": "ccabf139",
"id": "0244771b",
"metadata": {},
"source": [
"## Transform TensorFlow Model structure\n",
Expand All @@ -179,7 +181,7 @@
" ├── variables.data-00000-of-00001\n",
" └── variables.index\n",
"```\n",
"For Triton, the model needs to be have the following structure:\n",
"For Triton, the model needs to have the following structure:\n",
"```\n",
"<model-name>\n",
"├── config.pbtxt\n",
Expand All @@ -198,7 +200,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "6bac71fb",
"id": "24f277d4",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -208,7 +210,7 @@
},
{
"cell_type": "markdown",
"id": "ec59ac19",
"id": "782a1ffa",
"metadata": {},
"source": [
"### Inspect the model using the `saved_model_cli`\n",
Expand All @@ -220,7 +222,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "0d5bf935",
"id": "e7658452",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -229,7 +231,7 @@
},
{
"cell_type": "markdown",
"id": "f5f90c83",
"id": "f86cf399",
"metadata": {},
"source": [
"### Create the config.pbtxt \n",
Expand All @@ -242,7 +244,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "2d71bced",
"id": "bb769e57",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -279,7 +281,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "8ec22ec1",
"id": "f62b2c6c",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -288,7 +290,7 @@
},
{
"cell_type": "markdown",
"id": "09df0acc",
"id": "85594834",
"metadata": {},
"source": [
"### Create the tar ball in the required Triton structure"
Expand All @@ -297,7 +299,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "847402ce",
"id": "706058b9",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -308,7 +310,7 @@
},
{
"cell_type": "markdown",
"id": "8e98334b",
"id": "a6f18435",
"metadata": {},
"source": [
"### Upload the new tar ball containing the Triton model structure to s3"
Expand All @@ -317,7 +319,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "84517edd",
"id": "6fab3787",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -326,7 +328,7 @@
},
{
"cell_type": "markdown",
"id": "bbcd4c68",
"id": "e3211424",
"metadata": {},
"source": [
"## Deploy model to SageMaker Endpoint\n",
Expand All @@ -340,7 +342,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "5ccba9d9",
"id": "a83ca640",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -357,7 +359,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "3f1a9592",
"id": "4a452e78",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -371,7 +373,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "9e6e2fda",
"id": "6af73bb6",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -385,7 +387,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "0a65c498",
"id": "d7ab780f",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -404,7 +406,9 @@
"}\n",
"runtime_sm_client = boto3.client(\"sagemaker-runtime\")\n",
"response = runtime_sm_client.invoke_endpoint(\n",
" EndpointName=endpoint_name, ContentType=\"application/octet-stream\", Body=json.dumps(payload)\n",
" EndpointName=endpoint_name,\n",
" ContentType=\"application/octet-stream\",\n",
" Body=json.dumps(payload),\n",
")\n",
"\n",
"predictions = json.loads(response[\"Body\"].read())[\"outputs\"][0][\"data\"]\n",
Expand All @@ -415,21 +419,21 @@
},
{
"cell_type": "markdown",
"id": "732def02",
"id": "0c4e881c",
"metadata": {},
"source": [
"## Clean up\n",
"Optionally delete the Real-time endpoint created"
"We strongly recommend to delete the Real-time endpoint created to stop incurring cost when finished with the example"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b89fac8",
"id": "74556045",
"metadata": {},
"outputs": [],
"source": [
"# sm_client.delete_endpoint(EndpointName=endpoint_name)"
"sm_client.delete_endpoint(EndpointName=endpoint_name)"
]
}
],
Expand Down

0 comments on commit 07cb67f

Please sign in to comment.