Creating_and_Evaluating_Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating and Evaluating Solutions \n",
    "\n",
    "To recap from the first notebook:\n",
    "\n",
    "For the most part the algorithms in Amazon Personalize look to solve different tasks explained here:\n",
    "\n",
    "1. HRNN & HRNN-Metadata - Personalization\n",
    "1. HRNN Coldstart - Personalization that promotes new conten\n",
    "1. Personalized-Ranking - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.\n",
    "1. SIMS(Similar Items) - Given one item, what other items are also interacted with by users.\n",
    "1. Popularity-Count - What items are most popular, if HRNN or HRNN-Metadata do not have an answer for the user you query, this is what is returned by default.\n",
    "\n",
    "\n",
    "No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:\n",
    "\n",
    "1. UserID - User who interacted\n",
    "1. ItemID - Item the user interacted with\n",
    "1. Timestamp - When did this interaction occur\n",
    "\n",
    "We also support event types and event values defined by:\n",
    "\n",
    "1. Event Type - Categorical label of an event (browse, purcahsed, rated, etc).\n",
    "1. Event Value - Something corresponding to event type that happened. Generally speaking we look to normalized between 0 and 1 for the values over the types. So if there are three phases to complete a transaction (clicked, added-to-cart, and purchased) there would be an event_value for each phase as 0.33, 0.66, 1.0 respectfully.\n",
    "\n",
    "In this particular exercise we will leave event_type and event_value ignored. They can come in handy later but are skipped for the initial POC. \n",
    "\n",
    "The previous notebooks covered:\n",
    "\n",
    "1. Selecting a dataset.\n",
    "1. Preparing interactions data for Personalize.\n",
    "1. Preparing item or user metadata for Personalize [Optional].\n",
    "1. Creating a Dataset Group.\n",
    "1. Creating and importing data into an Interactions dataset within the dataset group.\n",
    "1. Creating and importing data into the metadata datasets [Optional].\n",
    "\n",
    "\n",
    "## Creating Solutions\n",
    "\n",
    "This nobeook will cover creating the following solutions:\n",
    "\n",
    "1. HRNN\n",
    "1. SIMS\n",
    "1. Personalized-Ranking\n",
    "\n",
    "After that the metrics will be explained and another notebook will showcase how to interact with the Solutions once they are deployed into a Campaign.\n",
    "\n",
    "The first step is to reload the imports and the stored variables from the previous notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the SDK to Personalize:\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Amazon Personalize a trained model is called a Solution, each Solution can have many specific versions that relate to a given volume of data when the model was trained.\n",
    "\n",
    "To begin we will list all the recipies that are supported, a recipie is an algorithm that has not been trained on your data yet. After listing you'll select one and use that to build your model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list_recipes_response = personalize.list_recipes()\n",
    "list_recipes_response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That is just a JSON representation of all of the algorithms that we have mentioned already. \n",
    "\n",
    "Next we will select a particular algorithm then build a model with it.\n",
    "\n",
    "### HRNN\n",
    "\n",
    "\n",
    "HRNN is one of the more advanced recommendation models that you can use and it allows for things like real-time updates of recommendations based on user behavior. It also tends to out perform other approaches like collaborative filtering. We will kick this job off first as it takes the longest to complete.\n",
    "\n",
    "#### Select Recipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "HRNN_recipe_arn = \"arn:aws:personalize:::recipe/aws-hrnn\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create and Wait for Solution\n",
    "First you will create the solution with the API, then you will create a version. \n",
    "\n",
    "Note the solution is just a label kind of identifier, you'll also need to create a version which is the actual trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hrnn_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-hrnn\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = HRNN_recipe_arn\n",
    ")\n",
    "\n",
    "hrnn_solution_arn = hrnn_create_solution_response['solutionArn']\n",
    "print(json.dumps(hrnn_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the Solution Version\n",
    "\n",
    "This process will actually take a while to complete, upwards of 25 minutes on. Normally there would be while loops to poll until the task is completed. However the task would block other cells from executing and the goal here is to create many models and deploy them quickly. Below there are instructions to viewing the progress in browser. After creating all of the solution versions go there and watch for updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hrnn_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = hrnn_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hrnn_solution_version_arn = hrnn_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(hrnn_create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SIMS\n",
    "\n",
    "\n",
    "SIMS is one of the longest running algorithms within Amazon for recommendation systems. A core use case for it is when you have one item and you want to recommend items that have been intereacted with in similar ways over your entire user base(not personalized). Sometimes this leads to recommending mostly popular items, so there is a hyperparameter that can be tweaked that will reduce the popular items in your results. \n",
    "\n",
    "Just as last time we start by selecting the recipie:\n",
    "\n",
    "#### Select Recipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## SIMS:\n",
    "SIMS_recipe_arn = \"arn:aws:personalize:::recipe/aws-sims\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create and Wait for Solution\n",
    "\n",
    "As with HRNN, start with the solution first.\n",
    "\n",
    "Note the solution is just a label kind of identifier, you'll also need to create a version which is the actual trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sims_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-sims\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = SIMS_recipe_arn\n",
    ")\n",
    "\n",
    "sims_solution_arn = sims_create_solution_response['solutionArn']\n",
    "print(json.dumps(sims_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the Solution Version\n",
    "\n",
    "This process will actually take a while to complete, upwards of 25 minutes on. Normally there would be while loops to poll until the task is completed. However the task would block other cells from executing and the goal here is to create many models and deploy them quickly. Below there are instructions to viewing the progress in browser. After creating all of the solution versions go there and watch for updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sims_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = sims_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sims_solution_version_arn = sims_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(sims_create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking\n",
    "\n",
    "Personalized Ranking is an interesting application of HRNN. Instead of just recommending what is most probable for your user in question, this algorithm takes in a user and a list of items as well. The items are then rendered back in the order of most probability for the user. The use case here is for filtering on genre for example, or when you have a broad collection that you would like better ordered for a particular user.\n",
    "\n",
    "#### Select Recipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Personalized-rerank\n",
    "rerank_recipe_arn = \"arn:aws:personalize:::recipe/aws-personalized-ranking\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create and Wait for Solution\n",
    "First you will create the solution with the API, then you will create a version. \n",
    "\n",
    "Note the solution is just a label kind of identifier, you'll also need to create a version which is the actual trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_create_solution_response = personalize.create_solution(\n",
    "    name = \"personalize-poc-rerank\",\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    recipeArn = rerank_recipe_arn\n",
    ")\n",
    "\n",
    "rerank_solution_arn = rerank_create_solution_response['solutionArn']\n",
    "print(json.dumps(rerank_create_solution_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create the Solution Version\n",
    "\n",
    "This process will actually take a while to complete, upwards of 25 minutes on. Normally there would be while loops to poll until the task is completed. However the task would block other cells from executing and the goal here is to create many models and deploy them quickly. Below there are instructions to viewing the progress in browser. After creating all of the solution versions go there and watch for updates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_create_solution_version_response = personalize.create_solution_version(\n",
    "    solutionArn = rerank_solution_arn\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_solution_version_arn = rerank_create_solution_version_response['solutionVersionArn']\n",
    "print(json.dumps(rerank_create_solution_version_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Viewing Solution Creation Status\n",
    "\n",
    "As promised, how to view the status updates in the console:\n",
    "\n",
    "* In another browser tab you should already have the AWS Console up from opening this notebook instance. \n",
    "* Switch to that tab and search at the top for the service `Personalize`, then go to that service page. \n",
    "* Click `View dataset groups`.\n",
    "* Click the name of your dataset group, most likely something with POC in the name.\n",
    "* Click `Solutions and recipes`.\n",
    "* You will now see a list of all of the solutions you created above. Click any one of them. \n",
    "* Note in `Solution versions` the job that is in progress. Once it is `Active` you solution is ready to be reviewed. It is also capable of being deployed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating Solutions\n",
    "\n",
    "After about an hour max the solutions should be ready for review. While they are in progress it is a good idea to cover the various algorithms and their behavior in depth. You'll have another lull period as the solutions are being deployed into campaigns as well, so you can split the material into 2 sections if that makes it easier. Also it can be a good time to discuss alternatives to how the data was fed into the system and what kind of results to expect from it.\n",
    "\n",
    "The firt step is to obtain the solutions metrics, API calls for each below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HRNN Metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hrnn_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = hrnn_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(hrnn_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above metrics tell us that at 5 items, we have less than a 1% chance (.8% literally) in a recommendation being a part of a user's interaction history (in the hold out phase from training and validation). This is clearly not a great model, but keep in mind instead of plays, all we had to go on is that they tagged a particular artist and that is it.\n",
    "\n",
    "Something else to note, the users' history is influenced by ANY recommendation model that is in place as your historical data is being collected. This often means that while your model probably won't be this bad with a customer or on your own data, it does bias the metrics to favor their existing solution. If you work to just push the offline metrics to match or exceed their existing solution you may just be making HRNN start to behave like whatever they were already using.\n",
    "\n",
    "This is a great time to have a conversation about AB testing and to think about the actual business outcomes they are looking to drive. From there you look to run small experiments with Personalize against their existing recommendation system and see over time how the AB test performs. If Personalize is winning then it is the time to filter more and more traffic to Personalize and campaigns within it. Over time the bias from the existing model will fade."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SIMS Metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sims_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = sims_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(sims_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we are seeing a slightly elivated precision at 5, a little over 1% at 1.04% this time. Effectively this is probably within the margin of error but given that no effort was made to mask popularity, it may just be returning super popular results that a large volume of users have interacted with in some way. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Personalized Ranking Metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rerank_solution_metrics_response = personalize.get_solution_metrics(\n",
    "    solutionVersionArn = rerank_solution_version_arn\n",
    ")\n",
    "\n",
    "print(json.dumps(rerank_solution_metrics_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storing Useful Variables]\n",
    "\n",
    "Before exiting this notebook, run the following cells to save off our version arns for use later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store hrnn_solution_version_arn\n",
    "%store sims_solution_version_arn\n",
    "%store rerank_solution_version_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just a quick comment on this one, here we see again a precision of near 1%, as this is based on HRNN, that is to be expected. \n",
    "\n",
    "You are now ready to move on to deploying and interacting with campaigns, continue by opening `Deploying_Campaigns_and_Interacting.ipynb`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}