Skip to content

Commit

Permalink
commit to address reviewer feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
@vllyakho
vllyakho committed Jul 5, 2022
1 parent 20d21ef commit 4664b43
Show file tree
Hide file tree
Showing 2 changed files with 96 additions and 49 deletions.
6 changes: 3 additions & 3 deletions ingest_data/sagemaker-keyspaces/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
# Train Machine Learning Models using Amazon Keyspaces as a Data Source
# Train Machine Learning Models using Amazon Keyspaces as a Data Source

Please read [Train machine learning models using Amazon Keyspaces as a data source](https://aws.amazon.com/blogs/machine-learning/train-machine-learning-models-using-amazon-keyspaces-as-a-data-source/) blog for more detailed instructions to run this notebook.


We will provide step-by-step instructions to use SageMaker to ingest customer data from Amazon Keyspaces and train a clustering model that would enable you to segment customers. This information can be used for targeted marketing, greatly improving your business KPI.
We provides step-by-step instructions to use SageMaker to ingest customer data from Amazon Keyspaces and train a clustering model that enables you to segment customers. This information can be used for targeted marketing, greatly improving your business KPI.

1. First, we install Sigv4 driver to connect to Amazon Keyspaces

Expand All @@ -13,7 +13,7 @@ We will provide step-by-step instructions to use SageMaker to ingest customer da
3. Next, we create new Keyspace ***blog_(yyyymmdd)*** and a new table ***online_retail***
3. Next, we download retail data about customers.
3. Next, we ingest retail data about customers into Keyspaces.
3. Next, we use a notebook available within SageMaker Studio to collect data from the Keyspaces database, and prepare data for training using KNN Algorithm. Most of our customers use SageMaker Studio for end-to-end development of ML Use Cases. They use this notebook as a starting point and customize it for their use case. Also, they are able to share this with other collaborators without requiring them to install any additional software.
3. Next, we use a notebook available within SageMaker Studio to collect data from the Keyspaces database, and prepare data for training using KNN Algorithm. Most of our customers use SageMaker Studio for end-to-end development of ML Use Cases. They use this notebook as a starting point and customize it for their use case. Also, they are able to share this with other collaborators without requiring them to install any additional software.
3. Next, we train the data for clustering.
3. When the training is completed, we can view the mapping between customers and their associated clusters.
3. And finally, we run a Cleanup Step to drop Keyspaces table to avoid future charges.
Expand Down
139 changes: 93 additions & 46 deletions ingest_data/sagemaker-keyspaces/SageMaker_Keyspaces_ml_example.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
"cells": [
{
"cell_type": "markdown",
"id": "5f11bc6e",
"id": "207aa954",
"metadata": {},
"source": [
"## Train Machine Learning Models using Amazon Keyspaces as a Data Source. \n",
"## Train Machine Learning Models using Amazon Keyspaces as a Data Source \n",
"\n",
"Contributors\n",
"- `Vadim Lyakhovich (AWS)`\n",
Expand All @@ -20,7 +20,7 @@
},
{
"cell_type": "markdown",
"id": "4ba54244",
"id": "2d10bdcf",
"metadata": {},
"source": [
"### Prerequisites\n",
Expand Down Expand Up @@ -56,7 +56,7 @@
},
{
"cell_type": "markdown",
"id": "3299688a",
"id": "29439d73",
"metadata": {},
"source": [
"In this notebook, \n",
Expand All @@ -67,10 +67,10 @@
"\n",
"2. Next, we establish a connection to Amazon Keyspaces \n",
"3. Next, we create a new `Keyspace ***blog_(yyyymmdd)***` and a new `table ***online_retail***`\n",
"3. Next, we will download retail data about customers.\n",
"3. Next, we will ingest retail data about customers into Keyspaces.\n",
"3. Next, we use a notebook available within SageMaker Studio to collect data from Keyspaces database, and prepare data for training using KNN Algorithm. Most of our customers use SageMaker Studio for end to end development of ML Use Cases. They could use this notebook as a base and customize it quickly for their use case. Additionally, they will be able to share this with other collaborators without requiring them to install any additional software. \n",
"3. Next, we will train the data for clustering.\n",
"3. Next, we download retail data about customers.\n",
"3. Next, we ingest retail data about customers into Keyspaces.\n",
"3. Next, we use a notebook available within SageMaker Studio to collect data from Keyspaces database, and prepare data for training using KNN Algorithm. Most of our customers use SageMaker Studio for end to end development of ML Use Cases. They could use this notebook as a base and customize it quickly for their use case. Additionally, the customers can share this with other collaborators without requiring them to install any additional software. \n",
"3. Next, we train the data for clustering.\n",
"3. After the training is complete, we can view the mapping between customer and their associated cluster.\n",
"3. And finally, Cleanup Step to drop Keyspaces table to avoid future charges. \n",
"\n"
Expand All @@ -79,7 +79,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "40527f71",
"id": "17cd92c7",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -128,7 +128,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "74968829",
"id": "aee7e72d",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -164,7 +164,7 @@
},
{
"cell_type": "markdown",
"id": "b94bca48",
"id": "e14d7fa9",
"metadata": {},
"source": [
"## Download Sample data \n",
Expand All @@ -177,7 +177,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "4cee4b1b",
"id": "c3aa9ea8",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -188,11 +188,10 @@
},
{
"cell_type": "markdown",
"id": "b2445623",
"id": "717151f6",
"metadata": {},
"source": [
"## In this step we will create a new ***blog_(yyyymmdd)*** keyspace and ***online_retail*** table\n",
"In this step we will create a new `Keyspace ***blog_yyymmdd***` and `table ***online_retail***`\n",
"#### In this step we create a new ***blog_(yyyymmdd)*** keyspace and ***online_retail*** table\n",
"\n",
"```\n",
"CREATE KEYSPACE IF NOT EXISTS blog_yyyymmdd\n",
Expand All @@ -217,7 +216,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "f80c6ada",
"id": "98ccf29c",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -252,7 +251,7 @@
},
{
"cell_type": "markdown",
"id": "dd5322a5",
"id": "ce87792a",
"metadata": {},
"source": [
"## Loading Data\n",
Expand All @@ -262,7 +261,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "8d8d53bd",
"id": "2ce1813a",
"metadata": {},
"outputs": [],
"source": [
Expand Down Expand Up @@ -340,42 +339,91 @@
},
{
"cell_type": "markdown",
"id": "7f27dfa2",
"id": "e2dc6a6a",
"metadata": {},
"source": [
"## ML Code"
]
},
{
"cell_type": "markdown",
"id": "c89529f1",
"id": "79e48035",
"metadata": {},
"source": [
"Now that we have data in Keyspace, let's read the data from Keyspace into the data frame. Once you have data into data frame you can perform the data cleanup to make sure it’s ready to train the modal. \n",
"Now that we have data in Keyspace, let's read the data from Keyspace into the data frame. Once you have data into data frame you can perform the data cleanup to make sure it’s ready to train the modal. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "433fbc4a",
"metadata": {},
"outputs": [],
"source": [
"# Reading Data from Keyspaces\n",
"r = session.execute(\"select * from \" + keyspaces_schema + \".online_retail\")\n",
"\n",
"df = DataFrame(r)\n",
"df.head(100)"
]
},
{
"cell_type": "markdown",
"id": "cdfa1734",
"metadata": {},
"source": [
"in this example, we use CQL to read records from the Keyspace table.\n",
"\n",
"In some ML use-cases, you may need to read the same data from the same Keyspaces table multiple times. In this case, we recommend to save your data into an Amazon S3 bucket to avoid incurring additional costs reading from Amazon Keyspaces. Depending on your scenario, you may also use Amazon EMR to ingest a very large Amazon S3 file into SageMaker."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "0e06c0f7",
"metadata": {},
"outputs": [],
"source": [
"## Code to save Python DataFrame to S3\n",
"import sagemaker\n",
"from io import StringIO # python3 (or BytesIO for python2)\n",
"\n",
"smclient = boto3.Session().client(\"sagemaker\")\n",
"sess = sagemaker.Session()\n",
"bucket = sess.default_bucket() # Set a default S3 bucket\n",
"print(bucket)\n",
"\n",
"sess = sagemaker.Session()\n",
"\n",
"In this example we will also group the data based on Recency, Frequency and Monetary value to generate RFM Matrix. Our business objective given the data set is to cluster the customers using this specific metric call RFM. The RFM model is based on three quantitative factors:\n",
"\n",
"csv_buffer = StringIO()\n",
"df.to_csv(csv_buffer)\n",
"s3_resource = boto3.resource(\"s3\")\n",
"s3_resource.Object(bucket, \"out/saved_online_retail.csv\").put(Body=csv_buffer.getvalue())"
]
},
{
"cell_type": "markdown",
"id": "e7b822ce",
"metadata": {},
"source": [
"In this example we also group the data based on Recency, Frequency and Monetary value to generate RFM Matrix. Our business objective given the data set is to cluster the customers using this specific metric call RFM. The RFM model is based on three quantitative factors:\n",
"\n",
" - Recency: How recently a customer has made a purchase.\n",
" - Frequency: How often a customer makes a purchase.\n",
" - Monetary Value: How much money a customer spends on purchases.\n",
"\n",
"RFM analysis numerically ranks a customer in each of these three categories, generally on a scale of 1 to 5 (the higher the number, the better the result). The “best” customer would receive a top score in every category. We’ll use pandas’s Quantile-based discretization function (qcut). It will help discretize values into equal-sized buckets based or based on sample quantiles. At end we see predicted cluster / segments for customers described like \"New Customers\", \"Hibernating\", \"Promising\" etc. "
"RFM analysis numerically ranks a customer in each of these three categories, generally on a scale of 1 to 5 (the higher the number, the better the result). The best customer receives a top score in every category. We are using pandas’s Quantile-based discretization function (qcut). It helps discretize values into equal-sized buckets based or based on sample quantiles. At end we see predicted cluster / segments for customers described like \"New Customers\", \"Hibernating\", \"Promising\" etc. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "37d10662",
"id": "5199e64a",
"metadata": {},
"outputs": [],
"source": [
"# Prepare Data\n",
"\n",
"r = session.execute(\"select * from \" + keyspaces_schema + \".online_retail\")\n",
"\n",
"df = DataFrame(r)\n",
"df.head(100)\n",
"\n",
"df.count()\n",
"df[\"description\"].nunique()\n",
"df[\"totalprice\"] = df[\"quantity\"] * df[\"price\"]\n",
Expand Down Expand Up @@ -453,14 +501,14 @@
},
{
"cell_type": "markdown",
"id": "50c8b2a7",
"id": "9ad3ab64",
"metadata": {},
"source": [
"Now that we have our final dataset, we will start our training. \n",
"Now that we have our final dataset, we can start our training. \n",
"\n",
"Here you will notice that we are doing data engineering by using a transform function that scales each feature to a given range. MinMaxScaler() fubction scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one in our case.\n",
"Here you notice that we are doing data engineering by using a transform function that scales each feature to a given range. `MinMaxScaler()` function scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one in our case.\n",
"\n",
"Next we do transform (analyzes the data to generate the coefficients) and fit (calculates the parameters/weights on training data) on the input data at a single time and converts the data points. This fit_transform() method used below is basically a combination of fit method and transform method, it is equivalent to fit(). transform(). \n",
"Next we do transform (analyzes the data to generate the coefficients) and fit (calculates the parameters/weights on training data) on the input data at a single time and converts the data points. This `fit_transform()` method used below is basically a combination of fit method and transform method, it is equivalent to `fit(). transform()`. \n",
"\n",
"\n",
"Next, the KMeans algorithm creates the clusters (customers grouped together based on various attributes in the data set). This cluster information (a.k.a segments) can be used for targeted marketing campaigns."
Expand All @@ -469,7 +517,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "d1bdcfaa",
"id": "ea020e9f",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -487,7 +535,7 @@
},
{
"cell_type": "markdown",
"id": "04ca83e8",
"id": "46ca68e4",
"metadata": {},
"source": [
"Let’s visualize the data to see how records are distributed in different clusters."
Expand All @@ -496,7 +544,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "1717f156",
"id": "e1258e4a",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -517,21 +565,20 @@
},
{
"cell_type": "markdown",
"id": "b151fb27",
"id": "e822c021",
"metadata": {},
"source": [
"## (Optional)\n",
"Next, we save the customer segments that have been identified by the ML model back to an Amazon Keyspaces table for targeted marketing. A batch job could read this data and run targeted campaigns to customers in specific segments."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "063f4aa5",
"id": "bc74f01d",
"metadata": {},
"outputs": [],
"source": [
"# Create ml_clustering_results table to store results\n",
"# Create ml_clustering_results table to store the results\n",
"createTable = \"\"\"CREATE TABLE IF NOT EXISTS %s.ml_clustering_results ( \n",
" run_id text,\n",
" segment int,\n",
Expand Down Expand Up @@ -572,17 +619,17 @@
},
{
"cell_type": "markdown",
"id": "0665dcb8",
"id": "2bd56924",
"metadata": {},
"source": [
"## Cleanup \n",
"Finally, we clean up the resources created during this tutorial to avoid incurring additional charges. So in this step we will drop the Keyspaces to prevent future charges"
"Finally, we clean up the resources created during this tutorial to avoid incurring additional charges. So in this step we drop the Keyspaces to prevent future charges"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "5e51f453",
"id": "3b76a063",
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -597,7 +644,7 @@
},
{
"cell_type": "markdown",
"id": "1f680e91",
"id": "039a2999",
"metadata": {},
"source": [
"It may take a few seconds to a minute to complete the deletion of keyspace and tables. When you delete a keyspace, the keyspace and all of its tables are deleted and you stop accruing charges from them.\n",
Expand All @@ -609,7 +656,7 @@
{
"cell_type": "code",
"execution_count": null,
"id": "29be2259",
"id": "d9de6153",
"metadata": {},
"outputs": [],
"source": []
Expand Down

0 comments on commit 4664b43

Please sign in to comment.