Validating_and_Importing_User_Item_Interaction_Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating and Importing User-Item-Interaction Data\n",
    "\n",
    "\n",
    "For the most part the algorithms in Amazon Personalize look to solve different tasks explained here:\n",
    "\n",
    "1. HRNN & HRNN-Metadata - Personalization\n",
    "1. HRNN Coldstart - Personalization that promotes new content\n",
    "1. Personalized-Ranking - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.\n",
    "1. SIMS(Similar Items) - Given one item, what other items are also interacted with by users.\n",
    "1. Popularity-Count - What items are most popular, if HRNN or HRNN-Metadata do not have an answer for the user you query, this is what is returned by default.\n",
    "\n",
    "\n",
    "No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:\n",
    "\n",
    "1. UserID - User who interacted\n",
    "1. ItemID - Item the user interacted with\n",
    "1. Timestamp - When did this interaction occur\n",
    "\n",
    "We also support event types and event values defined by:\n",
    "\n",
    "1. Event Type - Categorical label of an event (browse, purcahsed, rated, etc).\n",
    "1. Event Value - Something corresponding to event type that happened. Generally speaking we look to normalized between 0 and 1 for the values over the types. So if there are three phases to complete a transaction (clicked, added-to-cart, and purchased) there would be an event_value for each phase as 0.33, 0.66, 1.0 respectfully.\n",
    "\n",
    "In this particular exercise we will leave event_type and event_value ignored. They can come in handy later but are skipped for the initial POC. \n",
    "\n",
    "## Chosing a Dataset or Data Source\n",
    "\n",
    "As we mentioned the user-item-iteraction data is key for getting started with the service. This means we need to look for use cases that generate that kind of data, a few common examples are:\n",
    "\n",
    "1. Video-on-Demand applications\n",
    "1. E-Commerce platforms\n",
    "1. Social-Media aggregators / platforms\n",
    "\n",
    "There are also a few guidelines for is the problem correctly sized for Personalize, the minimum recommendations are below:\n",
    "\n",
    "* Authenticated users\n",
    "* At least 50 users\n",
    "* At least 100 items\n",
    "* At least 2 dozen interactions for each. \n",
    "\n",
    "Most of the time this is easily attainable, and if you are low in one category, you can often make it up by having a larger number in the other. \n",
    "\n",
    "Generally speaking your data will not arrive in a perfect form for this application and will take some modifications to get structured correctly. This notebook looks to guide you through all of that. \n",
    "\n",
    "To begin with, we are going to use the Last.FM dataset found [here](https://grouplens.org/datasets/hetrec-2011/). This data fits our guidelines with a large number for users, items, and interactions. \n",
    "\n",
    "Next you will use the cells below to create a folder for the example data as well as download the dataset for analysis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = \"poc_data\"\n",
    "!mkdir $data_dir\n",
    "!cd $data_dir && wget http://files.grouplens.org/datasets/hetrec2011/hetrec2011-lastfm-2k.zip\n",
    "!cd $data_dir && unzip hetrec2011-lastfm-2k.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At present not much is known about the data other than we seem to have many .dat files and a readme. Opening the readme will tell us about the overall structure of this data. This is a step you probably can skip with custom data unless the data source is coming from an external team. \n",
    "\n",
    "Note the data does not seem to be encoded with utf-8 for some reason so it is recommended that you open a terminal and `cat` the file so that you can see the output as Jupyter Lab/Notebooks do not render text documents that are not utf-8.\n",
    "\n",
    "\n",
    "Performing that yielded some interesting stats about the data:\n",
    "\n",
    "```\n",
    "---------------\n",
    "Data statistics\n",
    "---------------\n",
    "\n",
    "    1892 users\n",
    "   17632 artists\n",
    "      \n",
    "   12717 bi-directional user friend relations, i.e. 25434 (user_i, user_j) pairs\n",
    "         avg. 13.443 friend relations per user\n",
    "         \n",
    "   92834 user-listened artist relations, i.e. tuples [user, artist, listeningCount]\n",
    "         avg. 49.067 artists most listened by each user\n",
    "         avg. 5.265 users who listened each artist\n",
    "            \n",
    "   11946 tags  \n",
    "   \n",
    "  186479 tag assignments (tas), i.e. tuples [user, tag, artist]\n",
    "         avg. 98.562 tas per user\n",
    "         avg. 14.891 tas per artist\n",
    "         avg. 18.930 distinct tags used by each user\n",
    "         avg. 8.764 distinct tags used for each artist\n",
    "\n",
    "-----\n",
    "```\n",
    "\n",
    "In this case we are focusing on the users, the artists, and the listening relations so we have 1892, 17632, and 92834 items to meet those, a great volume of data for getting started.\n",
    "\n",
    "The focus on this notebook is again the interactions so we should look to find some data that supports it.\n",
    "\n",
    "\n",
    "```\n",
    "-----\n",
    "Files\n",
    "-----\n",
    "            \n",
    "   * artists.dat\n",
    "   \n",
    "        This file contains information about music artists listened and tagged by the users.\n",
    "   \n",
    "   * tags.dat\n",
    "   \n",
    "        This file contains the set of tags available in the dataset.\n",
    "\n",
    "   * user_artists.dat\n",
    "   \n",
    "        This file contains the artists listened by each user.\n",
    "        \n",
    "        It also provides a listening count for each [user, artist] pair.\n",
    "\n",
    "   * user_taggedartists.dat - user_taggedartists-timestamps.dat\n",
    "   \n",
    "        These files contain the tag assignments of artists provided by each particular user.\n",
    "        \n",
    "        They also contain the timestamps when the tag assignments were done.\n",
    "   \n",
    "   * user_friends.dat\n",
    "   \n",
    "        These files contain the friend relations between users in the database.\n",
    "     \n",
    "-----------\n",
    "Data format\n",
    "-----------\n",
    "\n",
    "   The data is formatted one entry per line as follows (tab separated, \"\\t\"):\n",
    "\n",
    "   * artists.dat\n",
    "   \n",
    "        id \\t name \\t url \\t pictureURL\n",
    "\n",
    "        Example:\n",
    "        707     Metallica       http://www.last.fm/music/Metallica      http://userserve-ak.last.fm/serve/252/7560709.jpg\n",
    "\n",
    "   * tags.dat\n",
    " \n",
    "        tagID \\t tagValue\n",
    "        1       metal\n",
    " \n",
    "   * user_artists.dat\n",
    "   \n",
    "        userID \\t artistID \\t weight\n",
    "        2       51      13883\n",
    "   \n",
    "   * user_taggedartists.dat\n",
    "  \n",
    "        userID \\t artistID \\t tagID \\t day \\t month \\t year\n",
    "        2       52      13      1       4       2009  \n",
    "  \n",
    "   * user_taggedartists-timestamps.dat\n",
    "\n",
    "        userID \\t artistID \\t tagID \\t timestamp\n",
    "        2       52      13      1238536800000\n",
    "\n",
    "   * user_friends.dat\n",
    "\n",
    "        userID \\t friendID\n",
    "        2       275\n",
    "\n",
    "```\n",
    "\n",
    "So right off the bad we now see a problem that although there is data supporting users interacting with just artists, for some reason we only have it stored as a weight not an actual timestamp. \n",
    "\n",
    "It does look like when a user tagged an artist there is a timestamp, so what if we make an assumption that tagging was a positive indicator and we use that data to get started? It seems like a solid approach for the POC, so that is what we are going to do now. \n",
    "\n",
    "The schema for the `user_taggedartists-timestamps.dat` is:\n",
    "\n",
    "| userID | artistID | tagID | timestamp     |\n",
    "|--------|----------|-------|---------------|\n",
    "| 2      | 52       | 13    | 1238536800000 |\n",
    "\n",
    "\n",
    "That looks pretty good for our base. Only the tagID needs to be removed. \n",
    "\n",
    "\n",
    "## Preparing Your Data\n",
    "\n",
    "The next thing to be done is to read the data with Pandas and confirm the data is in a good state, and save it to a CSV where it is ready to be used with Amazon Personalize"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the Pandas library as well as a few other data science tools in order to inspect the information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from time import sleep\n",
    "import subprocess\n",
    "import pandas as pd\n",
    "import json\n",
    "import time\n",
    "import pprint\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.dates import DateFormatter\n",
    "import matplotlib.dates as mdates\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next open the file with Pandas and take a look at the contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data = pd.read_csv(data_dir + '/user_taggedartists-timestamps.dat')\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well that did not work so well, looks like the tab delimiter needs to be specified, attempt 2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data = pd.read_csv(data_dir + '/user_taggedartists-timestamps.dat', delimiter='\\t')\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data looks really good here but lets get some extra insights on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now there is clearly a range of values for all of the columns which is great, the last one to be mindful of is that the timestamp should be in Unix Epoch format. You can learn more about the format here: https://en.wikipedia.org/wiki/Unix_time\n",
    "\n",
    "Let us grab an arbitrary column and convert it to a datetime and confirm that it feels like a reasonable value for the historical data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = original_data.iloc[50]['timestamp']\n",
    "print(arb_time_stamp)\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this particular value it rendered a year of 41,132... a bit into the future for us, so somehow we parsed it incorrectly. Attempt number 2...\n",
    "\n",
    "JavaScript records time in milliseconds and this is a collection of data from a web application, so divide by 1000 first and see what is returned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = arb_time_stamp/1000\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feb 2009 feels completely reasonable so now move forward by transforming each row in the dataframe in the same way."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "original_data.head(5)\n",
    "original_data.timestamp = original_data.timestamp / 1000\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to see if the timestamps hold correctly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "arb_time_stamp = original_data.iloc[50]['timestamp']\n",
    "print(arb_time_stamp)\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks exactly like we are after, now we can drop the tagID column. First make a copy of the df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df = original_data.copy()\n",
    "interactions_df = interactions_df[['userID', 'artistID', 'timestamp']]\n",
    "interactions_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.astype({'timestamp': 'int64'}).dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Personalize has default column names of users, items and timestamp so now we will replace our data set with the correct values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.rename(columns = {'userID':'USER_ID', 'artistID':'ITEM_ID', \n",
    "                              'timestamp':'TIMESTAMP'}, inplace = True) \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point the data is ready to go, we just need to save it as a CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_filename = \"interactions.csv\"\n",
    "interactions_df.to_csv((data_dir+\"/\"+interactions_filename), index=False, float_format='%.0f')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating Dataset Groups and the Interactions Dataset\n",
    "\n",
    "The highest level of isolation and abstraction with Amazon Personalize is a Dataset Group. Information stored within one of these has no impact on any other dataset group or models created from one. This allows you to run many experiments and is part of how we keep your models private and fully trained only on your data. \n",
    "\n",
    "Before importing the data prepared earlier, there needs to be a dataset group and a dataset added to it that handles the interactions.\n",
    "\n",
    "Dataset Groups can house the following types of information:\n",
    "\n",
    "* User-Item-Interactions\n",
    "* Event Streams ( Real time Interactions )\n",
    "* User Metadata\n",
    "* Item Metadata\n",
    "\n",
    "The cells below will create the dataset group and the dataset for interactions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now validate that your environment can communicate successfully with Amazon Personalize, the lines below do just that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the SDK to Personalize:\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Dataset Group"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_dataset_group_response = personalize.create_dataset_group(\n",
    "    name = \"personalize-poc-lastfm\"\n",
    ")\n",
    "\n",
    "dataset_group_arn = create_dataset_group_response['datasetGroupArn']\n",
    "print(json.dumps(create_dataset_group_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wait for Dataset Group to Have ACTIVE Status\n",
    "\n",
    "Before we can use the Dataset Group in any items below it must be active, execute the cell below and wait for it to show active."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_group_response = personalize.describe_dataset_group(\n",
    "        datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    status = describe_dataset_group_response[\"datasetGroup\"][\"status\"]\n",
    "    print(\"DatasetGroup: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Dataset\n",
    "\n",
    "First define a schema for the interactions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_schema = schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Interactions\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"TIMESTAMP\",\n",
    "            \"type\": \"long\"\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "create_schema_response = personalize.create_schema(\n",
    "    name = \"personalize-poc-lastfm-interactions\",\n",
    "    schema = json.dumps(interactions_schema)\n",
    ")\n",
    "\n",
    "schema_arn = create_schema_response['schemaArn']\n",
    "print(json.dumps(create_schema_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now create a dataset with that schema."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset_type = \"INTERACTIONS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"personalize-poc-lastfm-ints\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = schema_arn\n",
    ")\n",
    "\n",
    "dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_dataset_arn = dataset_arn"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring S3 and IAM \n",
    "\n",
    "\n",
    "Amazon Personalize will need an S3 bucket to act as the source of your data, as well as IAM roles for accessing it. The code below will set all that up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now using the metada stored on this instance of a SageMaker Notebook determine the region we are operating in. If you are using a Jupyter Notebook outside of SageMaker simply define region as the string that indicates the region you would like to use for Forecast and S3.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "session = boto3.Session(region_name=region)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(region)\n",
    "s3 = boto3.client('s3')\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "bucket_name = account_id + \"personalizepoc\"\n",
    "print(bucket_name)\n",
    "if region != \"us-east-1\":\n",
    "    s3.create_bucket(Bucket=bucket_name, CreateBucketConfiguration={'LocationConstraint': region})\n",
    "else:\n",
    "    s3.create_bucket(Bucket=bucket_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Attach Policy to S3 Bucket\n",
    "Amazon Personalize needs to be able to read the content of your S3 bucket that you created earlier. The lines below will do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s3 = boto3.client(\"s3\")\n",
    "\n",
    "policy = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Id\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"Service\": \"personalize.amazonaws.com\"\n",
    "            },\n",
    "            \"Action\": [\n",
    "                \"s3:*Object\",\n",
    "                \"s3:ListBucket\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::{}\".format(bucket_name),\n",
    "                \"arn:aws:s3:::{}/*\".format(bucket_name)\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Personalize Role\n",
    "Also Amazon Personalize needs the ability to assume Roles in AWS in order to have the permissions to execute certain tasks, the lines below grant that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "iam = boto3.client(\"iam\")\n",
    "\n",
    "role_name = \"PersonalizeRolePOC\"\n",
    "assume_role_policy_document = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "          \"Effect\": \"Allow\",\n",
    "          \"Principal\": {\n",
    "            \"Service\": \"personalize.amazonaws.com\"\n",
    "          },\n",
    "          \"Action\": \"sts:AssumeRole\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "create_role_response = iam.create_role(\n",
    "    RoleName = role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)\n",
    ")\n",
    "\n",
    "# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes \"personalize\" or \"Personalize\" \n",
    "# if you would like to use a bucket with a different name, please consider creating and attaching a new policy\n",
    "# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role\n",
    "policy_arn = \"arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess\"\n",
    "iam.attach_role_policy(\n",
    "    RoleName = role_name,\n",
    "    PolicyArn = policy_arn\n",
    ")\n",
    "\n",
    "# Now add S3 support\n",
    "iam.attach_role_policy(\n",
    "    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',\n",
    "    RoleName=role_name\n",
    ")\n",
    "time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "print(role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Upload to S3\n",
    "\n",
    "Before Personalize can import the data, it needs to be in S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload Interactions File\n",
    "interactions_file_path = data_dir + \"/\" + interactions_filename\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(interactions_filename).upload_file(interactions_file_path)\n",
    "interactions_s3DataPath = \"s3://\"+bucket_name+\"/\"+interactions_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing the Interactions Data\n",
    "\n",
    "Earlier you created the DatasetGroup and Dataset to house your information, now you will execute an import job that will load the data from S3 into Amazon Personalize for usage building your model.\n",
    "\n",
    "#### Create Dataset Import Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"personalize-poc-import1\",\n",
    "    datasetArn = interactions_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket_name, interactions_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Wait for Dataset Import Job to Have ACTIVE Status\n",
    "It can take a while before the import job completes, please wait until you see that it is active below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n",
    "        datasetImportJobArn = dataset_import_job_arn\n",
    "    )\n",
    "    status = describe_dataset_import_job_response[\"datasetImportJob\"]['status']\n",
    "    print(\"DatasetImportJob: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that the dataset import is active you are ready to start building models with SIMS, Personalized-Ranking, Popularity-Count, and HRNN. Work will continue in other notebooks. Run the cell below before moving on to store a few values for usage in the next notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store interactions_dataset_arn\n",
    "%store dataset_group_arn\n",
    "%store bucket_name\n",
    "%store role_arn\n",
    "%store role_name\n",
    "%store data_dir"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}