intermediate_textmining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intermediate TextMining with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " By: Dr. Eric Godat and Dr. Rob Kalescky\n",
    " \n",
    " Acknowledgements: Garrett Moore, Jaymie Ruddock\n",
    "\n",
    " Adapted from: [Ultimate Guide to deal with Text Data (Using Python)](https://www.analyticsvidhya.com/blog/2018/02/the-different-methods-deal-text-data-predictive-python/)\n",
    " \n",
    " Natural Language Toolkit: [Documentation](http://www.nltk.org/)\n",
    " \n",
    " Reference Text: [Natural Language Processing with Python](http://www.nltk.org/book/)\n",
    " \n",
    " \n",
    " *This notebook follows the **Introduction to Working with Text Data** lesson. Please consider looking over that lesson if you have not already*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the basic libraries we will use in for data manipulation (pandas) and math functions (numpy). We will add more libraries as we need them.\n",
    "\n",
    "As a best practice, it is a good idea to load all your libraries in a single cell at the top of a notebook, however for the purposes of this tutorial we will load some now and more as we go."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_format ='retina'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import glob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load a data file into a pandas DataFrame.\n",
    "\n",
    "This tutorial was designed around using sets of data you have yourselves in a form like a CSV, TSV, or TXT file.  Feel free to use any set of data, but for now we will use a dataset created from scraping this [Multilingual Folktale Database](http://www.mftd.org/).\n",
    "\n",
    "This file is a CSV filetype, common for text data, but your data may also be stored as TSV's, TXT's, or other file types.  This will slightly change how you read from Pandas, but the concept is largely the same for the different filetypes.  Just keep this in mind when you see references to CSV.\n",
    "\n",
    "To proceed, you will need to have this file downloaded and in the same folder as this notebook. Alternatively you can put the full path to the file.  Typically, your program will look for the file with the name you specified in the folder that contains your program unless you give the program a path to follow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "glob.glob(\"*.csv\") #Allows us to check the files in the current directory, if folktales.csv is not here, then we need to give the path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = 'folktales.csv' #If you need to put the path to the file, do so here.\n",
    "data = pd.read_csv(filename)#if your filetype is not CSV, you may need to add \" , sep = 'separating_character_here' \" after the filename.  This may require extra manipulation, so be careful.\n",
    "#an alternative, depending on your filetype and situation, may be read_table as opposed to read_csv\n",
    "data.rename(columns={'Unnamed: 0':'Index'},inplace=True) # Rename our index column\n",
    "data.head() # We will use the .head() attribute to limit the amount of the DataFrame is displayed on screen. It is not necessary for computation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see all the information available to us from the file in the form of a Pandas DataFrame. For the remainder of this tutorial, we will focus primarily on the full text of each data chunk, which we will name the *Story* column.  With your data set this is likely to be something very different, so feel free to call is something else."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counting Words and Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first bit of analysis we might want to do is to count the number of words in one piece of data. To do this we will add a column called *wordcount* and write an operation that applies a function to every row of the column.\n",
    "\n",
    "Unpacking this piece of code, *len(str(x).split(\" \")*, tells us what is happening.\n",
    "\n",
    "For the content of cell *x*, convert it to a string, *str()*, then split that string into pieces at each space, *split()*.\n",
    "\n",
    "The result of that is a list of all the words in the text and then we can count the length of that list, *len()*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data['wordcount'] = data['Story'].apply(lambda x: len(str(x).split(\" \")))\n",
    "data[['Story','wordcount']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do something similar to count the number of characters in the data chunk, including spaces. If you wanted to exclude whitespaces, you could take the list we made above, join it together and count the length of the resulting string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = data.fillna(\"No Information Provided\") #If some of our data is missing, this will replace the blank entries. This is only necessary in some cases"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['char_count'] = data['Story'].str.len() ## this also includes spaces, to do it without spaces, you could use something like this: \"\".join()\n",
    "data[['Story','char_count']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to calculate the average word length in the data.\n",
    "\n",
    "Let's define a function that will do that for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def avg_word(sentence):\n",
    "    words = sentence.split()\n",
    "    return (sum(len(word) for word in words)/len(words))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now apply that function to all the data chunks and save that in a new column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['avg_word'] = data['Story'].apply(lambda x: avg_word(x))\n",
    "data[['Story','avg_word']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then sort by the average word length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[['Story','avg_word']].sort_values(by='avg_word', ascending=True).head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Processing Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A major component of doing analysis on text is the cleaning of the text prior to the analysis.\n",
    "\n",
    "Though this process destroys some elements of the text (sentence structure, for example), it is often necessary in order to describe a text analytically. Depending on your choice of cleaning techniques, some elements might be preserved better than others if that is of importance to your analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning Up Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This series of steps aims to clean up and standardize the text itself. This generally consists of removing common elements such as stopwords and punctuation but can be expanded to more detailed removals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lowercase"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we enforce that all of the text is lowercase. This makes it easier to match cases and sort words.\n",
    "\n",
    "Notice we are assigning our modified column back to itself. This will save our modifications to our DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['Story'] = data['Story'].apply(lambda x: \" \".join(x.lower() for x in x.split()))\n",
    "data['Story'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we remove all punctuation from the data. This allows us to focus on the words only as well as assist in matching."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['Story'] = data['Story'].str.replace('[^\\w\\s]','')\n",
    "data['Story'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stopwords are words that are commonly used and do little to aid in the understanding of the content of a text. There is no universal list of stopwords and they vary on the style, time period and media from which your text came from.  Typically, people choose to remove stopwords from their data, as it adds extra clutter while the words themselves provide little to no insight as to the nature of the data.  For now, we are simply going to count them to get an idea of how many there are.\n",
    "\n",
    "For this tutorial, we will use the standard list of stopwords provided by the Natural Language Toolkit python library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop = stopwords.words('english')\n",
    "data['Story'] = data['Story'].apply(lambda x: \" \".join(x for x in x.split() if x not in stop))\n",
    "data['Story'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove Frequent Words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to catch common words that might have slipped through the stopword removal, we can build out a list of the most common words remaining in our text.\n",
    "\n",
    "Here we have built a list of the 10 most common words. Some of these words might actually be relevant to our analysis so it is important to be careful with this method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = pd.Series(' '.join(data['Story']).split()).value_counts()[:10]\n",
    "freq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now follow the same procedure with which we removed stopwords to remove the most frequent words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "freq = list(freq.index)\n",
    "data['Story'] = data['Story'].apply(lambda x: \" \".join(x for x in x.split() if x not in freq))\n",
    "data['Story'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lemmatization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lemmatization is often a more useful approach than stemming because it leverages an understanding of the word itself to convert the word back to its root word. However, this means lemmatization is less aggressive than stemming (probably a good thing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#import nltk\n",
    "#nltk.download('wordnet')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from textblob import Word\n",
    "data['Story'] = data['Story'].apply(lambda x: \" \".join([Word(word).lemmatize() for word in x.split()]))\n",
    "data['Story'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point we have a several options for cleaning and structuring our text data. The next section will focus on more advanced ways to study text analytically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Text Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This section focuses on more complex methods of analyzing textual data. We will continue to work with our same DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## N-grams"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "N-grams are combinations of multiple words as they appear in the text. The N refers to the number of words captured in the list. N-grams with N=1 are referred unigrams and are just a nested list of all the words in the text. Following a similar pattern, bigrams (N=2), trigrams (N=3), etc. can be used.\n",
    "\n",
    "N-grams allow you to capture the structure of the text which can be very useful. For instance, counting the number of bigrams where \"said\" was preceded by \"he\" vs \"she\" could give you an idea of the gender breakdown of speakers in a text. However, if you make your N-grams too long, you lose the ability to make comparisons.\n",
    "\n",
    "Another concern, especially in very large data sets, is that the memory storage of N-grams scales with N (bigrams are twice as large as unigrams, for example) and the time to process the N-grams can balloon dramatically as well.\n",
    "\n",
    "All that being said, we would suggest focusing on bigrams and trigrams as useful analysis tools."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "from textblob import TextBlob\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "n_grams = TextBlob(data['Story'][1]).ngrams(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "characters=[]\n",
    "for i in ['wolf', 'mouse', 'rabbit', 'children']:\n",
    "     characters.append(lemmatizer.lemmatize(i))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for n in n_grams:\n",
    "    if n[1] in characters:\n",
    "        print(n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk import ngrams\n",
    "from collections import Counter\n",
    "\n",
    "ngram_counts = Counter(ngrams(data['Story'][1].split(), 2))\n",
    "for n in ngram_counts.most_common(10):\n",
    "    print(n)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Term Frequency is a measure of how often a term appears in a document. There are different ways to define this but the simplest is a raw count of the number of times each term appears.\n",
    "\n",
    "There are other ways of defining this including a true term frequency and a log scaled definition. All three have been implemented below but the default will the raw count definition, as it matches with the remainder of the definitions in this tutorial.\n",
    "\n",
    "|Definition|Formula|\n",
    "|---|---|\n",
    "|Raw Count|$$f_{t,d}$$|\n",
    "|Term Frequency|$$\\frac{f_{t,d}}{\\sum_{t'\\in d}f_{t',d}}$$|\n",
    "|Log Scaled|$$\\log(1+f_{t,d})$$|\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Raw Count Definition\n",
    "tf1 = (data['Story'][0:5]).apply(lambda x: pd.value_counts(x.split(\" \"))).sum(axis = 0).reset_index()\n",
    "\n",
    "## Term Frequency Definition\n",
    "#tf1 = (data['Story'][0:5]).apply(lambda x: (pd.value_counts(x.split(\" \")))/len(x.split(\" \"))).sum(axis = 0).reset_index() \n",
    "\n",
    "## Log Scaled Definition\n",
    "#tf1 = (data['Story'][0:10]).apply(lambda x: 1.0+np.log(pd.value_counts(x.split(\" \")))).sum(axis = 0).reset_index() \n",
    "\n",
    "tf1.columns = ['words','tf']\n",
    "tf1.sort_values(by='tf', ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inverse Document Frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inverse Document Frequency is a measure of how common or rare a term is across multiple documents. That gives a measure of how much weight that term carries.\n",
    "\n",
    "For a more concrete analogy of this, imagine a room full of NBA players; here a 7 foot tall person wouldn't be all that shocking. However if you have a room full of kindergarten students, a 7 foot tall person would be a huge surprise.\n",
    "\n",
    "The simplest and standard definition of Inverse Document Frequency is to take the logarithm of the ratio of the number of documents containing a term to the total number of documents.\n",
    "\n",
    "$$-\\log\\frac{n_t}{N} = \\log\\frac{N}{n_t}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i,word in enumerate(tf1['words']):\n",
    "    tf1.loc[i, 'idf'] = np.log(data.shape[0]/(len(data[data['Story'].str.contains(word)])))\n",
    "\n",
    "tf1[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Term Frequency – Inverse Document Frequency ([TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Term Frequency – Inverse Document Frequency (TF-IDF) is a composite measure of both Term Frequency and Inverse Document Frequency.\n",
    "\n",
    "From [Wikipedia](https://en.wikipedia.org/wiki/Tf%E2%80%93idf):\n",
    "\"A high weight in TF–IDF is reached by a high term frequency (in the given document) and a low document frequency of the term in the whole collection of documents; the weights hence tend to filter out common terms\"\n",
    "\n",
    "More concisely, a high TD-IDF says that a word is very important in the documents in which it appears.\n",
    "\n",
    "There are a few weighting schemes for TF-IDF. Here we use scheme (1).\n",
    "\n",
    "|Weighting Scheme|Document Term Weight|\n",
    "|---|---|\n",
    "|(1)|$$f_{t,d}\\cdot\\log\\frac{N}{n_t}$$|\n",
    "|(2)|$$1+\\log(f_{t,d})$$|\n",
    "|(3)|$$(1+\\log(f_{t,d}))\\cdot\\log\\frac{N}{n_t}$$|"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tf1['tfidf'] = tf1['tf'] * tf1['idf']\n",
    "tf1.sort_values(by='tfidf', ascending=False)[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is worth noting that the *sklearn* library has the ability to directly calculate a TD-IDF matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "tfidf = TfidfVectorizer(max_features=1000, lowercase=True, analyzer='word',\n",
    " stop_words= 'english',ngram_range=(1,1))\n",
    "data_vect = tfidf.fit_transform(data['Story'])\n",
    "\n",
    "data_vect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\", category=FutureWarning) # This suppresses a warning from scikit learn that they are going to update their code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One thing we can look at is how similar two texts are. This has a practical use when looking for plagiarism, but can also be used to compare author's styles. To do this there are a few ways we can measure similarity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to set up our two sets of texts. Here we have the ability to choose the size of our sets and whether we want the first n texts from our full data set or just a random sample. Keep in mind that if you want to use two dissimilar sets, you won't have a control value (something x itself)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "size=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## First n by First n\n",
    "set1 = data[[\"Index\",'Story','Title']][:size]\n",
    "set2 = data[[\"Index\",'Story','Title']][:size]\n",
    "\n",
    "## Random X Itself\n",
    "#set1 = data[[\"Index\",'Story','Title']].sample(size)\n",
    "#set2 = set1\n",
    "\n",
    "## First n by Random\n",
    "#set1 = data[[\"Index\",'Story','Title']][:size]\n",
    "#set2 = data[[\"Index\",'Story','Title']].sample(size)\n",
    "\n",
    "## Random X Random\n",
    "#set1 = data[[\"Index\",'Story','Title']].sample(size)\n",
    "#set2 = data[[\"Index\",'Story','Title']].sample(size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#To let us do a \"cross join\": Every row in one gets matched to all rows in the other\n",
    "set1['key']=1 \n",
    "set2['key']=1\n",
    "\n",
    "similarity = pd.merge(set1, set2, on ='key', suffixes=('_1', '_2')).drop(\"key\", 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Jaccard Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first is using a metric called the [Jaccard Index](https://en.wikipedia.org/wiki/Jaccard_index). This is just taking the intersection of two sets of things (in our case, words or n-grams) and dividing it by the union of those sets. This gives us a metric for understanding how the word usage compares but doesn't account for repeated words since the union and intersections just take unique words. One advantage though is that we can easily extend the single word similarity to compare bi-grams and other n-grams if we want to examine phrase usage.\n",
    "\n",
    "$$S_{J}(A,B)=\\frac{A \\cap B}{A \\cup B}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def jaccard(row, n=1):\n",
    "    \n",
    "    old=row['Story_1']\n",
    "    new=row['Story_2']\n",
    "    \n",
    "    old_n_grams = [tuple(el) for el in TextBlob(old).ngrams(n)]\n",
    "    new_n_grams = [tuple(el) for el in TextBlob(new).ngrams(n)]\n",
    "        \n",
    "    union = list(set(old_n_grams) | set(new_n_grams))\n",
    "    intersection = list(set(old_n_grams) & set(new_n_grams))\n",
    "    \n",
    "    lu = len(union)\n",
    "    li = len(intersection)\n",
    "    \n",
    "    return (li/lu,li,lu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in [1,2]: # Add values to the list for the n-gram you're interested in\n",
    "    similarity['Jaccard_Index_for_{}_grams'.format(i)]=similarity.apply(lambda x: jaccard(x,i)[0],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity['Intersection']=similarity.apply(lambda x: jaccard(x)[1],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity['Union']=similarity.apply(lambda x: jaccard(x)[2],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cosine Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The second metric we can use is [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity), however there is a catch here. Cosine similarity requires a vector for each word so we make a choice here to use term frequency. You could choose something else, inverse document frequency or tf-idf would both be good choices. Cosine similarity with a term frequency vector gives us something very similar to the Jaccard Index but accounts for word repetition. This makes it better for tracking word importance between two texts.\n",
    "\n",
    "$$S_{C}(v_1,v_2)=cos(\\theta)=\\frac{v_1\\cdot v_2}{||v_1||\\times||v_2||}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cosine_sim(str1,str2): \n",
    "    vectors = [t for t in get_vectors([str1,str2])]\n",
    "    return cosine_similarity(vectors)\n",
    "    \n",
    "def get_vectors(slist):\n",
    "    text = [t for t in slist]\n",
    "    vectorizer = CountVectorizer(text)\n",
    "    vectorizer.fit(text)\n",
    "    return vectorizer.transform(text).toarray()\n",
    "\n",
    "def cosine(row):\n",
    "    \n",
    "    old=row['Story_1']\n",
    "    new=row['Story_2']\n",
    "    \n",
    "    return get_cosine_sim(old,new)[0,1]   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity['Cosine_Similarity']=similarity.apply(lambda x: cosine(x),axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize Similarity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric = 'Cosine_Similarity'\n",
    "#metric = 'Jaccard_Index_for_1_grams'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "index = list(similarity['Index_1'].unique())\n",
    "columns = list(similarity['Index_2'].unique())\n",
    "df = pd.DataFrame(0, index=index, columns=columns)\n",
    "\n",
    "for i in df.index:\n",
    "    sub = similarity[(similarity['Index_1']==i)]\n",
    "    for col in df.columns:\n",
    "        df.loc[i,col]=sub[sub['Index_2']==col][metric].iloc[0]\n",
    "        \n",
    "plt.pcolor(df)\n",
    "plt.yticks(np.arange(0.5, len(df.index), 1), df.index)\n",
    "plt.xticks(np.arange(0.5, len(df.columns), 1), df.columns)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "similarity[['Title_1','Index_1','Title_2','Index_2',metric]].sort_values(by=metric,ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## [Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Bag of Words](https://en.wikipedia.org/wiki/Bag-of-words_model) a way to represent text based on the idea that similar texts will contain similar vocabulary. There is a lot to this model and we provide merely a simple implementation of it here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "bow = CountVectorizer(max_features=1000, lowercase=True, ngram_range=(1,1),analyzer = \"word\")\n",
    "data_bow = bow.fit_transform(data['Story'])\n",
    "data_bow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment is a way of measuring the overall positivity or negativity in a given text.\n",
    "\n",
    "To do this we will use the built in sentiment function in the *TextBlob* package. This function will return the polarity and subjectivity scores for each data chunk."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['Story'][:5].apply(lambda x: TextBlob(x).sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Focusing on the polarity score, we are able to see the overall sentiment of each data chunk. The closer to 1 the more positive and the closer to -1 the more negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data['sentiment'] = data['Story'].apply(lambda x: TextBlob(x).sentiment[0] )\n",
    "data[['Story','sentiment']].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have textted the sentiment scores for the first 10 chunks.\n",
    "\n",
    "Notice they tend to be positive but not exceedingly so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "plot_1 = data[['sentiment']][:10].plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have sorted and textted all of the sentiment scores for the chunks in our database.\n",
    "\n",
    "We can clearly see that most of the text data is positive but not overwhelmingly so (as seen by the long tail of the distribution). However, the parts that are negative tend to be more polarized than the positive ones (a shorter tail and sharper peak)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_2 = data[['sentiment']].sort_values(by='sentiment', ascending=False).plot(kind='bar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using TF-IDF and Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is significantly more advanced than the rest of the tutorial. This takes the TF-IDF matrix and applies a [k-means clustering algorithm](https://en.wikipedia.org/wiki/K-means_clustering). This groups the texts into clusters of similar terms from the TF-IDF matrix. This algorithm randomly seeds X \"means\", the values are then clustered into the nearest mean. The centroid of the values in each cluster then becomes the new mean and the process repeats until a convergence is reached."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.manifold import TSNE\n",
    "\n",
    "groups = 10\n",
    "\n",
    "num_clusters = groups\n",
    "num_seeds = groups\n",
    "max_iterations = 300\n",
    "labels_color_map = {\n",
    "    0: '#20b2aa', 1: '#ff7373', 2: '#ffe4e1', 3: '#005073', 4: '#4d0404',\n",
    "    5: '#ccc0ba', 6: '#4700f9', 7: '#f6f900', 8: '#00f91d', 9: '#da8c49'\n",
    "}\n",
    "pca_num_components = 2\n",
    "tsne_num_components = 2\n",
    "\n",
    "# calculate tf-idf of texts\n",
    "tfidf = TfidfVectorizer(max_features=1000, lowercase=True, analyzer='word',stop_words= 'english',ngram_range=(1,1))\n",
    "tf_idf_matrix = tfidf.fit_transform(data['Story'])\n",
    "\n",
    "# create k-means model with custom config\n",
    "clustering_model = KMeans(\n",
    "    n_clusters=num_clusters,\n",
    "    max_iter=max_iterations,\n",
    "    precompute_distances=\"auto\",\n",
    "    n_jobs=-1\n",
    ")\n",
    "\n",
    "labels = clustering_model.fit_predict(tf_idf_matrix)\n",
    "#print(labels)\n",
    "\n",
    "X = tf_idf_matrix.todense()\n",
    "\n",
    "# ----------------------------------------------------------------------------------------------------------------------\n",
    "\n",
    "reduced_data = PCA(n_components=pca_num_components).fit_transform(X)\n",
    "# print(reduced_data)\n",
    "\n",
    "import matplotlib.patches as mpatches\n",
    "legendlist=[mpatches.Patch(color=labels_color_map[key],label=str(key))for key in labels_color_map.keys()]\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "for index, instance in enumerate(reduced_data):\n",
    "    #print(instance, index, labels[index])\n",
    "    pca_comp_1, pca_comp_2 = reduced_data[index]\n",
    "    color = labels_color_map[labels[index]]\n",
    "    ax.scatter(pca_comp_1, pca_comp_2, c=color)\n",
    "plt.legend(handles=legendlist)\n",
    "plt.show()\n",
    "\n",
    "\n",
    "\n",
    "# t-SNE plot\n",
    "#embeddings = TSNE(n_components=tsne_num_components)\n",
    "#Y = embeddings.fit_transform(X)\n",
    "#plt.scatter(Y[:, 0], Y[:, 1], cmap=plt.cm.Spectral)\n",
    "#plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tfidf_test = tf1.sort_values(by='tfidf', ascending=False)[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_groups = np.transpose([labels,data['Title'],data['Story Type']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are the titles of the texts in each cluster. Keep in mind that each time you run the algorithm, the randomness in generating the initial means will result in different clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(len(title_groups)):\n",
    "    data.loc[i,'Group'] = title_groups[i][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grp=1\n",
    "data[data['Group']==grp]['Title'].sample(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(groups):\n",
    "    print(\"\")\n",
    "    print(\"#### {} ###\".format(i))\n",
    "    print(\"\")\n",
    "    for el in title_groups:\n",
    "        if el[0]==i:\n",
    "            print(\"{}\".format(el[1]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.4"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": false,
   "sideBar": false,
   "threshold": 6,
   "toc_cell": true,
   "toc_section_display": "block",
   "toc_window_display": false
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}