N_Gram_Text_Analyzer_for_SEO.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": " N-Gram Text Analyzer for SEO.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyO0fFrO57OjBZvun2HYNRFw",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LanceMeister/N_Gram_Text_Analyzer_for_SEO/blob/main/N_Gram_Text_Analyzer_for_SEO.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Requirements and Assumptions\n",
        "\n",
        "- Python 3 is installed and basic Python syntax understood\n",
        "- Access to a Linux installation (I recommend Ubuntu) or Google Colab.\n",
        "- Google Knowledge Base API key\n",
        "- You have a body of text to analyze\n",
        "## Import and Install Modules\n",
        "\n",
        "- nltk: NLP module handles the n-graming and word type tagging\n",
        "  - nltk.download(‘punkt’): functions for n-gramming\n",
        "  - nltk.download(‘averaged_perceptron_tagger’): functions for word type tagging\n",
        "- pandas: for storing and displaying the results\n",
        "- requests: for making API calls to Keyword Sufer API\n",
        "- json: for processing the Keyword Surfer API response\n",
        "First, let’s install the nltk module which you won’t like have already.  If you are using Google Colab put an exclamation mark at the beginning."
      ],
      "metadata": {
        "id": "PsqYgHDVhLHa"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zA7-IxS0ew0T",
        "cellView": "form"
      },
      "outputs": [],
      "source": [
        "#@title\n",
        "!pip3 install nltk\n",
        "!pip install --upgrade google-api-python-client"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "# -*- coding: utf-8 -*-\n",
        "import nltk\n",
        "nltk.download('punkt')\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "nltk.download('stopwords')\n",
        "from nltk.util import ngrams\n",
        "import requests\n",
        "import json\n",
        "import pandas as pd"
      ],
      "metadata": {
        "id": "J6yKtH0Xe5S0",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Build N-Grams from Provided Text\n",
        "\n",
        "We’re going to start off with a few functions. I decided to use functions because my app will process 1, 2, 3-gram tables when this script as is, will only process one of them of your choosing. You can automate at the end for all 3 by using a simple while loop. This first function breaks the text into what n-gram you choose later on.\n",
        "\n",
        "1. Send the text data into the nltk ngram function that returns a list of those n-grams. It takes two parameters. The first is a tokenized version of that text which it processes in the word_tokenize function first and the second parameter is whatever whole number n-gram you want.\n",
        "2. We join all the n-grams that are returned into a single list\n",
        "3. For Unigrams we want to tag each tokenized word and filter out anything that isn’t a noun or action verb (present participle)\n",
        "4. Return the list back to the main script"
      ],
      "metadata": {
        "id": "HYubEgFehzHm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "def extract_ngrams(data, num):\n",
        "  n_grams = ngrams(nltk.word_tokenize(data), num)\n",
        "  gram_list = [ ' '.join(grams) for grams in n_grams]\n",
        "  \n",
        "  \n",
        "  query_tokens = nltk.pos_tag(gram_list)\n",
        "  query_tokens = [x for x in query_tokens if x[1] in ['NN','NNS','NNP','NNPS','VBG','VBN']]\n",
        "  query_tokens = [x[0] for x in query_tokens]\n",
        "\n",
        "  return query_tokens"
      ],
      "metadata": {
        "id": "lcaBWlZye8kz",
        "cellView": "form"
      },
      "execution_count": 5,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Detect Entities from N-Grams\n",
        "\n",
        "Next, we tackle the function that will take the list of n-grams and pass it to the Google Knowledge Graph (GKG) API to check if it’s an entity.\n",
        "\n",
        "1. Create a variable to store the n-grams that end up as entities and a variable for your GKG API key.\n",
        "2.  Loop through all the n-grams in the list\n",
        "3. Make a request to GKG using n-gram, API key, and limiting to first result found (highest score, most likely)\n",
        "4.  Load API JSON response into a JSON object to parse\n",
        "5.  Parse JSON object for “type”, aka entity category, and resultScore, aka their relevancy confidence metric. Some requests are not entities and will break the script so we use a Try/Except method. If the Except is triggered, we make a score of 0 and a label of none.\n",
        "6. In the JSON data, the entity labels are in a list because entities often have more than one label or category. We loop over this list and create a string delineated list. In the end, we remove the last hanging comma and create a space between the comma and the next label.\n",
        "7. If a label doesn’t equal none and has a resultScore of greater than 500 let’s build a list for that n-gram that stores the n-gram, its score, and its labels. Then we append that list to a master list to store all the entity data. So we’ll have a list of lists. We then return that list of lists back to the main script."
      ],
      "metadata": {
        "id": "OKwJm2LzfHmB"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "def kg(keywords):\n",
        "\n",
        "  kg_entities = []\n",
        "  apikey=''\n",
        "\n",
        "  for x in keywords:\n",
        "    url = f'https://kgsearch.googleapis.com/v1/entities:search?query={x}&key={apikey}&limit=1&indent=True'\n",
        "    payload = {}\n",
        "    headers= {}\n",
        "    response = requests.request(\"GET\", url, headers=headers, data = payload)\n",
        "    data = json.loads(response.text)\n",
        "\n",
        "    try:\n",
        "      getlabel = data['itemListElement'][0]['result']['@type']\n",
        "      score = round(float(data['itemListElement'][0]['resultScore']))\n",
        "    except:\n",
        "      score = 0\n",
        "      getlabel = ['none']\n",
        "\n",
        "    labels = \"\"\n",
        "    \n",
        "    for item in getlabel:\n",
        "      labels += item + \",\"\n",
        "    labels = labels[:-1].replace(\",\",\", \")\n",
        "    \n",
        "    if labels != ['none'] and score > 500:\n",
        "      kg_subset = []\n",
        "      kg_subset.append(x)\n",
        "      kg_subset.append(score)\n",
        "      kg_subset.append(labels)\n",
        "\n",
        "      kg_entities.append(kg_subset)\n",
        "  return kg_entities"
      ],
      "metadata": {
        "id": "D7-k7s-BfAjP",
        "cellView": "form"
      },
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Gather Entity Search Metrics\n",
        "\n",
        "For our final function, we build our API call to Keyword Surfer to get volume, CPC, and competition score. Then we build the dataframe with all of our information.\n",
        "\n",
        " 1. Create a list with only the entity types.\n",
        " 2. Create a list with only the entity names.\n",
        " 3. Convert the entity names list to a string so we can feed it to the URL API.\n",
        " 4. Build the Keyword Surfer API call using the keyword string-list. Please do not abuse this API or we may find it blocked at some point.\n",
        " 5. Return the JSON response to seo_data variable.\n",
        " 6. Build the empty dataframe which we’ll use to store the data in later.\n",
        " 7. Loop through the entities we fed to the API.\n",
        " 8. Some entities won’t have data. We detect and assign a value manually for CPC and competition.\n",
        " 9. We build a dictionary object using the data for each entity and then append it to the dataframe we created earlier.\n",
        " 10. We format the dataframe with alignment and sorting by Volume and then return the dataframe to the main script."
      ],
      "metadata": {
        "id": "E_rToI6FfS_0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "def surfer(entities):\n",
        "  entities_type = [x[2] for x in entities]\n",
        "  entities = [x[0] for x in entities]\n",
        "  keywords = json.dumps(entities)\n",
        "\n",
        "  url2 = 'https://db2.keywordsur.fr/keyword_surfer_keywords?country=de&keywords=' + keywords\n",
        "  response2 = requests.get(url2,verify=True)\n",
        "  seo_data = json.loads(response2.text)\n",
        "\n",
        "  d = {'Keyword': [], 'Volume': [], 'CPC':[], 'Competition':[], 'Entity Types':[]}\n",
        "  df = pd.DataFrame(data=d)\n",
        "  counter=0\n",
        "\n",
        "  for word in seo_data:\n",
        "\n",
        "    if seo_data[word][\"cpc\"] == '':\n",
        "      seo_data[word][\"cpc\"] = 0.0\n",
        "    \n",
        "    if seo_data[word][\"competition\"] == '':\n",
        "      seo_data[word][\"competition\"] = 0.0\n",
        "\n",
        "    new = {\"Keyword\":word,\"Volume\":str(seo_data[word][\"search_volume\"]),\"CPC\":\"$\"+str(round(float(seo_data[word][\"cpc\"]),2)),\"Competition\":str(round(float(seo_data[word][\"competition\"]),4)),'Entity Types':entities_type[counter]}\n",
        "    df = df.append(new, ignore_index=True)\n",
        "    counter +=1\n",
        "\n",
        "  df.style.set_properties(**{'text-align': 'left'})\n",
        "  df.sort_values(by=['Volume'], ascending=True)\n",
        "  return df"
      ],
      "metadata": {
        "id": "EbFZPxr1fI4o",
        "cellView": "form"
      },
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Provide Text and Call Functions\n",
        "\n",
        "We arrive now at the actual start of the script where the data variable should contain the text you want to analyze.  We clean the text a little to be free of some pesky punctuation that sometimes breaks analysis. Certainly, there is a more efficient way of handling it, but this works fine for now. Feel free to add more replacements."
      ],
      "metadata": {
        "id": "zQqP2Rjcfc3y"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "spreadsheetName=\" N-Gram Text Analyzer for SEO\" #@param {type:\"string\"}\n",
        "worksheetName='unigram'\n",
        "data = \"\"#@param {type:\"string\"}\n",
        "data = data.replace(',','')\n",
        "data = data.replace('.','')\n",
        "data = data.replace(';','')\n"
      ],
      "metadata": {
        "id": "bHlWT48_fYkO",
        "cellView": "form"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here we send the data from above to the first function we created that processes for ngrams. The second parameter number is the number of ngram you want to process.\n",
        "\n",
        "1 = unigram\n",
        "2 = bigram\n",
        "3 = trigram …\n",
        "Then we send the ngram lists to the Google Knowledge Graph API for entity detection. Lastly, we send the entities to the Keyword Surfer API for metrics, and finally, we display the dataframe."
      ],
      "metadata": {
        "id": "zjb9p5CvgzHY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1 = unigram "
      ],
      "metadata": {
        "id": "V8T-l89aoCE_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "keywords = extract_ngrams(data, 1)\n",
        "entities = kg(keywords)\n",
        "df = surfer(entities)\n",
        "df\n"
      ],
      "metadata": {
        "id": "n9OWm6ZLgnwv",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1 = unigram (German)"
      ],
      "metadata": {
        "id": "w46PbGaJ81lJ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Run this code when anaylzing German text to remove unwanted \"stop words\" from the table"
      ],
      "metadata": {
        "id": "8DJIDwJw9Ap0"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title\n",
        "from nltk.corpus import stopwords\n",
        "stopWords = set(stopwords.words('german'))\n",
        "# print(str(stopWords))\n",
        "df_de = df[~df.Keyword.isin(stopWords)]\n",
        "df_de"
      ],
      "metadata": {
        "cellView": "form",
        "id": "IU9MozfN5q_i"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2 = bigram "
      ],
      "metadata": {
        "id": "NUC7R7V1sCn5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title \n",
        "keywords = extract_ngrams(data, 2)\n",
        "entities = kg(keywords)\n",
        "df2 = surfer(entities)\n",
        "df2"
      ],
      "metadata": {
        "cellView": "form",
        "id": "FAE0XrJ9sMmT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 3 = trigram "
      ],
      "metadata": {
        "id": "xD0QMcVBsfAG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#@title \n",
        "keywords = extract_ngrams(data, 3)\n",
        "entities = kg(keywords)\n",
        "df3 = surfer(entities)\n",
        "df3"
      ],
      "metadata": {
        "id": "sF8zXjersmny",
        "cellView": "form"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j4XLpVUFVcpC"
      },
      "source": [
        "# Populating our Google Sheet"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NzLG2zCXXq6q",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "#https://pypi.org/project/gspread-pandas/\n",
        "#https://github.com/aiguofer/gspread-pandas/pull/47\n",
        "#https://gspread-pandas.readthedocs.io/en/latest/getting_started.html"
      ],
      "execution_count": 42,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LGGKD7nzXt18",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "!pip install gspread-pandas\n",
        "!pip install --upgrade google-auth[reauth]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3i2wGQnzyb8k",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "from google.colab import auth\n",
        "auth.authenticate_user()\n",
        "import google.auth\n",
        "import gspread"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tljyUghFyrVx",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "from gspread_pandas import Spread, Client\n",
        "import google.auth\n",
        "\n",
        "creds, project = google.auth.default()\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6Wgu5tlFynPB",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "spread = Spread(spreadsheetName, worksheetName, creds=creds)\n",
        "spread\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7GdQhApjzLGi",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "# Display available worksheets\n",
        "spread.sheets"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DydKTAfLzOx9",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "# Save DataFrame to worksheet 'SEO Analyzed Body Text', create it first if it doesn't exist\n",
        "spread.df_to_sheet(df, index=False, sheet='unigram', start='A1', replace=True)\n",
        "spread.df_to_sheet(df2, index=False, sheet='bigram', start='A1', replace=True)\n",
        "spread.df_to_sheet(df3, index=False, sheet='trigram', start='A1', replace=True)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}