Barbie_Retrieval_Augmented_Question_Answering_(RAQA)_Assignment (Assignment).ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/AI-Maker-Space/LLM-Ops-Vault/blob/main/Week%201/First%20Session/Barbie_Retrieval_Augmented_Question_Answering_(RAQA)_Assignment%20(Completed).ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PLi09z_3qW8O"
      },
      "source": [
        "# Questioning Barbie Reviews with RAQA (Retrieval Augmented Question Answering)\n",
        "\n",
        "In the following notebook, you are tasked with creating a system that answers questions based on information found in reviews of the 2023 Barbie movie.\n",
        "\n",
        "## Build 🏗️\n",
        "\n",
        "There are 3 main tasks in this notebook:\n",
        "\n",
        "1. Obtain and parse reviews from a review website\n",
        "2. Create a Vectorstore from the reviews\n",
        "3. Create a `RetrievalQA` chain \n",
        "\n",
        "## Ship 🚢\n",
        "\n",
        "Create a Hugging Face Space that hosts your application.\n",
        "\n",
        "## Share 🚀\n",
        "\n",
        "Make a social media post about your final application."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BBraMxjrwOx"
      },
      "source": [
        ">### Why RAQA and not RAG?\n",
        ">If we look at the original [paper](https://arxiv.org/abs/2005.11401), we find that RAG is a fairly specific and well defined term that isn't exactly the same as \"retrieve context, feed context to model in the prompt\".\n",
        ">For that reason, we're making the decision to delineate between \"actual\" RAG, and Retrieval Augmented Question Answering - which is not a well defined phrase."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zcL8585DsZML"
      },
      "source": [
        "### Pre-task Work\n",
        "\n",
        "All we really need to do to get started is to get our prerequisites!\n",
        "\n",
        "We'll be leveraging `langchain`, `openai`, and `pinecone` today.\n",
        "\n",
        "Check out the docs:\n",
        "- [LangChain](https://docs.langchain.com/docs/)\n",
        "- [OpenAI](https://github.com/openai/openai-python)\n",
        "- [Pinecone](https://docs.pinecone.io/docs/overview)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "vQa6rjDLqPCI"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U openai langchain \"pinecone-client[grpc]\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1wF8_ODX5h-P",
        "outputId": "f3395c72-029b-4e49-f0b9-aba7e3c2195f"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "import getpass\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Open AI API Key:\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"PINECONE_API_KEY\"] = getpass.getpass(\"Pinecone API Key:\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"PINECONE_ENV\"] = getpass.getpass(\"Pinecone Environment:\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FLrL342RtWxf"
      },
      "source": [
        "### Task 1: Data Preparation\n",
        "\n",
        "In this task we'll be collecting, and then parsing, our data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zh8QTtMhzY_g"
      },
      "source": [
        "#### Scraping IMDB Reviews of Barbie\n",
        "\n",
        "We'll use some Selenium based trickery to get the reviews we need to make our application.\n",
        "\n",
        "Check out the docs here:\n",
        "- [Selenium](https://www.selenium.dev/documentation/)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "PsGPV2zHuCj2"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U requests"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uCNCZdzrw81B",
        "outputId": "f74cc223-af93-491a-e92e-5d7ee1355d49"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U scrapy selenium"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You will need to install the `chromium-chromedriver` in order to use the method presented. \n",
        "\n",
        "`!apt install chromium-chromedriver` will install the chromedriver. \n",
        "\n",
        "You may have to use your terminal if you receive an error related to `sudo`. In that case, please use (in your terminal) `sudo apt install chromium-chromedriver`.\n",
        "\n",
        "Otherwise, the `.csv` is provided and can be loaded through `pandas` if you're experiencing issues relating to the web-scraping portion of the notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "Niq6DKMOwPIn"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "from scrapy.selector import Selector\n",
        "from selenium import webdriver\n",
        "from selenium.webdriver.common.by import By\n",
        "from selenium.webdriver.common.keys import Keys\n",
        "import time\n",
        "from tqdm import tqdm\n",
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "k9AA_f-DxGyv"
      },
      "outputs": [],
      "source": [
        "chrome_options = webdriver.ChromeOptions()\n",
        "chrome_options.add_argument('--headless')\n",
        "chrome_options.add_argument('--no-sandbox')\n",
        "chrome_options.add_argument('--disable-dev-shm-usage')\n",
        "driver = webdriver.Chrome(options=chrome_options)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "IsjYL7K_y1ip"
      },
      "outputs": [],
      "source": [
        "url = \"https://www.imdb.com/title/tt1517268/reviews/?ref_=tt_ov_rt\"\n",
        "driver.get(url)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "XPGysdwuy190"
      },
      "outputs": [],
      "source": [
        "sel = Selector(text = driver.page_source)\n",
        "review_counts = sel.css('.lister .header span::text').extract_first().replace(',','').split(' ')[0]\n",
        "more_review_pages = int(int(review_counts)/25)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eGSJ0vuDy4D2",
        "outputId": "5bd75470-a74a-4b2c-ba76-a9025d71c79b"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  0%|          | 0/54 [00:00<?, ?it/s]"
          ]
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 54/54 [00:03<00:00, 17.92it/s]\n"
          ]
        }
      ],
      "source": [
        "for i in tqdm(range(more_review_pages)):\n",
        "    try:\n",
        "        css_selector = 'load-more-trigger'\n",
        "        driver.find_element(By.ID, css_selector).click()\n",
        "    except:\n",
        "        pass"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "c46cAciBxr2C",
        "outputId": "d5a8737f-6162-434c-ef82-81317490960d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 100/100 [00:00<00:00, 198.29it/s]\n"
          ]
        }
      ],
      "source": [
        "rating_list = []\n",
        "review_date_list = []\n",
        "review_title_list = []\n",
        "author_list = []\n",
        "review_list = []\n",
        "review_url_list = []\n",
        "error_url_list = []\n",
        "error_msg_list = []\n",
        "reviews = driver.find_elements(By.CSS_SELECTOR, 'div.review-container')\n",
        "\n",
        "for d in tqdm(reviews):\n",
        "    try:\n",
        "        sel2 = Selector(text = d.get_attribute('innerHTML'))\n",
        "        try:\n",
        "            rating = sel2.css('.rating-other-user-rating span::text').extract_first()\n",
        "        except:\n",
        "            rating = np.NaN\n",
        "        try:\n",
        "            review = sel2.css('.text.show-more__control::text').extract_first()\n",
        "        except:\n",
        "            review = np.NaN\n",
        "        try:\n",
        "            review_date = sel2.css('.review-date::text').extract_first()\n",
        "        except:\n",
        "            review_date = np.NaN\n",
        "        try:\n",
        "            author = sel2.css('.display-name-link a::text').extract_first()\n",
        "        except:\n",
        "            author = np.NaN\n",
        "        try:\n",
        "            review_title = sel2.css('a.title::text').extract_first()\n",
        "        except:\n",
        "            review_title = np.NaN\n",
        "        try:\n",
        "            review_url = sel2.css('a.title::attr(href)').extract_first()\n",
        "        except:\n",
        "            review_url = np.NaN\n",
        "        rating_list.append(rating)\n",
        "        review_date_list.append(review_date)\n",
        "        review_title_list.append(review_title)\n",
        "        author_list.append(author)\n",
        "        review_list.append(review)\n",
        "        review_url_list.append(review_url)\n",
        "    except Exception as e:\n",
        "        error_url_list.append(url)\n",
        "        error_msg_list.append(e)\n",
        "review_df = pd.DataFrame({\n",
        "    'Review_Date':review_date_list,\n",
        "    'Author':author_list,\n",
        "    'Rating':rating_list,\n",
        "    'Review_Title':review_title_list,\n",
        "    'Review':review_list,\n",
        "    'Review_Url':review_url\n",
        "    })"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 424
        },
        "id": "W9A0C1tbyfuh",
        "outputId": "8a333f0d-5af6-42f0-f623-c35dbc3170bf"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Review_Date</th>\n",
              "      <th>Author</th>\n",
              "      <th>Rating</th>\n",
              "      <th>Review_Title</th>\n",
              "      <th>Review</th>\n",
              "      <th>Review_Url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>21 July 2023</td>\n",
              "      <td>LoveofLegacy</td>\n",
              "      <td>6</td>\n",
              "      <td>Beautiful film, but so preachy\\n</td>\n",
              "      <td>Margot does the best with what she's given, bu...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>26 July 2023</td>\n",
              "      <td>aherdofbeautifulwildponies</td>\n",
              "      <td>6</td>\n",
              "      <td>A Hot Pink Mess\\n</td>\n",
              "      <td>Before making Barbie (2023),</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>22 July 2023</td>\n",
              "      <td>imseeg</td>\n",
              "      <td>7</td>\n",
              "      <td>3 reasons FOR seeing it and 1 reason AGAINST.\\n</td>\n",
              "      <td>The first reason to go see it:</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31 July 2023</td>\n",
              "      <td>ramair350</td>\n",
              "      <td>10</td>\n",
              "      <td>As a guy I felt some discomfort, and that's o...</td>\n",
              "      <td>As much as it pains me to give a movie called ...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>22 July 2023</td>\n",
              "      <td>Natcat87</td>\n",
              "      <td>6</td>\n",
              "      <td>Too heavy handed\\n</td>\n",
              "      <td>As a woman that grew up with Barbie, I was ver...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>95</th>\n",
              "      <td>25 July 2023</td>\n",
              "      <td>kimberlyely-27550</td>\n",
              "      <td>10</td>\n",
              "      <td>Everyone watch this\\n</td>\n",
              "      <td>Barbie</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>96</th>\n",
              "      <td>23 July 2023</td>\n",
              "      <td>sandrachungate</td>\n",
              "      <td>6</td>\n",
              "      <td>Great for the first 30 minutes, then not\\n</td>\n",
              "      <td>The trailers and marketing for the movie were ...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>21 July 2023</td>\n",
              "      <td>michelle-kissler</td>\n",
              "      <td>5</td>\n",
              "      <td>Barbie, Swing and a Miss\\n</td>\n",
              "      <td>This film had the potential to become a cult c...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>98</th>\n",
              "      <td>20 July 2023</td>\n",
              "      <td>mavigangi</td>\n",
              "      <td>10</td>\n",
              "      <td>A colorful love letter to human kind\\n</td>\n",
              "      <td>I cried, I laughed, I dance and I did sing too...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>99</th>\n",
              "      <td>28 July 2023</td>\n",
              "      <td>mohamed-elbehairy</td>\n",
              "      <td>6</td>\n",
              "      <td>Had some good moments but that's it\\n</td>\n",
              "      <td>I found the movie to be refreshing, but it's f...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>100 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "     Review_Date                      Author Rating  \\\n",
              "0   21 July 2023                LoveofLegacy      6   \n",
              "1   26 July 2023  aherdofbeautifulwildponies      6   \n",
              "2   22 July 2023                      imseeg      7   \n",
              "3   31 July 2023                   ramair350     10   \n",
              "4   22 July 2023                    Natcat87      6   \n",
              "..           ...                         ...    ...   \n",
              "95  25 July 2023           kimberlyely-27550     10   \n",
              "96  23 July 2023              sandrachungate      6   \n",
              "97  21 July 2023            michelle-kissler      5   \n",
              "98  20 July 2023                   mavigangi     10   \n",
              "99  28 July 2023           mohamed-elbehairy      6   \n",
              "\n",
              "                                         Review_Title  \\\n",
              "0                    Beautiful film, but so preachy\\n   \n",
              "1                                   A Hot Pink Mess\\n   \n",
              "2     3 reasons FOR seeing it and 1 reason AGAINST.\\n   \n",
              "3    As a guy I felt some discomfort, and that's o...   \n",
              "4                                  Too heavy handed\\n   \n",
              "..                                                ...   \n",
              "95                              Everyone watch this\\n   \n",
              "96         Great for the first 30 minutes, then not\\n   \n",
              "97                         Barbie, Swing and a Miss\\n   \n",
              "98             A colorful love letter to human kind\\n   \n",
              "99              Had some good moments but that's it\\n   \n",
              "\n",
              "                                               Review  \\\n",
              "0   Margot does the best with what she's given, bu...   \n",
              "1                       Before making Barbie (2023),    \n",
              "2                      The first reason to go see it:   \n",
              "3   As much as it pains me to give a movie called ...   \n",
              "4   As a woman that grew up with Barbie, I was ver...   \n",
              "..                                                ...   \n",
              "95                                             Barbie   \n",
              "96  The trailers and marketing for the movie were ...   \n",
              "97  This film had the potential to become a cult c...   \n",
              "98  I cried, I laughed, I dance and I did sing too...   \n",
              "99  I found the movie to be refreshing, but it's f...   \n",
              "\n",
              "                        Review_Url  \n",
              "0   /review/rw9221648/?ref_=tt_urv  \n",
              "1   /review/rw9221648/?ref_=tt_urv  \n",
              "2   /review/rw9221648/?ref_=tt_urv  \n",
              "3   /review/rw9221648/?ref_=tt_urv  \n",
              "4   /review/rw9221648/?ref_=tt_urv  \n",
              "..                             ...  \n",
              "95  /review/rw9221648/?ref_=tt_urv  \n",
              "96  /review/rw9221648/?ref_=tt_urv  \n",
              "97  /review/rw9221648/?ref_=tt_urv  \n",
              "98  /review/rw9221648/?ref_=tt_urv  \n",
              "99  /review/rw9221648/?ref_=tt_urv  \n",
              "\n",
              "[100 rows x 6 columns]"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "review_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9gHLLYflzw_f"
      },
      "source": [
        "Let's save this `pd.DataFrame` as a `.csv` to our local session (this will be terminated when you terminate the Colab session) so we can leverage it in LangChain!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "N9FDhwbNz64p"
      },
      "outputs": [],
      "source": [
        "review_df.to_csv(\"./barbie_dg.csv\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {},
      "outputs": [],
      "source": [
        "data = review_df"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JtBD1H8ezNLO"
      },
      "source": [
        "#### Data Parsing\n",
        "\n",
        "Now that we have our data - let's go ahead and set up some tools to parse it into a more usable format for LangChain!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_CTameZZ0r76"
      },
      "source": [
        "Our reviews might contain a lot of information, and in order to ensure they don't exceed the context window of our model and to allow us to include a few reviews as context for each query - let's construct a system to \"chunk\" our data into smaller pieces.\n",
        "\n",
        "We'll be leveraging the `RecursiveCharacterTextSplitter` for this task today.\n",
        "\n",
        "While splitting our text seems like a simple enough task - getting this correct/incorrect can have massive downstream impacts on your application's performance.\n",
        "\n",
        "You can read the docs here:\n",
        "- [RecursiveCharacterTextSplitter](https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter)\n",
        "\n",
        "> ### HINT:\n",
        ">It's always worth it to check out the LangChain source code if you're ever in a bind - for instance, if you want to know how to transform a set of documents, check it out [here](https://github.com/langchain-ai/langchain/blob/5e9687a196410e9f41ebcd11eb3f2ca13925545b/libs/langchain/langchain/text_splitter.py#L268C18-L268C18)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "id": "uEgcUVtl00Xm"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "\n",
        "text_splitter = RecursiveCharacterTextSplitter(\n",
        "    chunk_size = 1000, ### YOUR CODE HERE, # the character length of the chunk\n",
        "    chunk_overlap = 200, ### YOUR CODE HERE, # the character length of the overlap between chunks\n",
        "    length_function = len ### YOUR CODE HERE, # the length function - in this case, character length (aka the python len() fn.)\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylT4jwmx3zCb"
      },
      "source": [
        "Now that we have our `RecursiveCharacterTextSplitter` set up - let's look at how it might split our source text. \n",
        "\n",
        "Keep in mind that the source text is split by `[\"\\n\\n\", \"\\n\", \" \", \"\"]` in that order.\n",
        "\n",
        "We know that each of the subheadings in our review `page_content` is separated by a newline character, so it will preferably chunk the review subheadings together. \n",
        "\n",
        "That's great! Let's move on to creating our index!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9cB0L_CN38W5"
      },
      "source": [
        "### Task 2: Creating an \"Index\"\n",
        "\n",
        "The term \"index\" is used largely to mean: Structured documents parsed into a useful format for querying, retrieving, and use in the LLM application stack."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GycdG53N4f9Z"
      },
      "source": [
        "#### Selecting Our VectorStore\n",
        "\n",
        "There are a number of different VectorStores, and a number of different strengths and weaknesses to each.\n",
        "\n",
        "In this notebook, we will be keeping it very simple by leveraging Pinecone's API Vector Database."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "T5o4vwSn4hfe",
        "outputId": "62bf250d-1c4f-49c3-b154-71e52d530ddf"
      },
      "outputs": [],
      "source": [
        "!pip install -q -U tiktoken"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's set up a Pinecone index using the methods provided in their [documentation](https://docs.pinecone.io/docs/langchain)!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {},
      "outputs": [],
      "source": [
        "import pinecone\n",
        "\n",
        "YOUR_API_KEY = os.environ[\"PINECONE_API_KEY\"]\n",
        "YOUR_ENV = os.environ[\"PINECONE_ENV\"]\n",
        "\n",
        "index_name = 'barbie-review-index-dg'\n",
        "\n",
        "pinecone.init(\n",
        "    api_key= YOUR_API_KEY,### YOUR CODE HERE,\n",
        "    environment= YOUR_ENV, ### YOUR CODE HERE\n",
        ")\n",
        "\n",
        "if index_name not in pinecone.list_indexes():\n",
        "    # we create a new index\n",
        "    pinecone.create_index(\n",
        "        name=index_name,\n",
        "        metric='cosine',\n",
        "        dimension= 1536 ### YOU CODE HERE - REMEMBER TO USE THE SAME DIMENSION AS THE EMBEDDING MODEL (text-embedding-ada-002)\n",
        "    )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we can connect to our index and view some statistics about it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 28,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 1536,\n",
              " 'index_fullness': 0.00117,\n",
              " 'namespaces': {'': {'vector_count': 117}},\n",
              " 'total_vector_count': 117}"
            ]
          },
          "execution_count": 28,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index = pinecone.GRPCIndex(index_name)\n",
        "\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CGU96p5R54Xz"
      },
      "source": [
        "We're going to be setting up our VectorStore with the OpenAI embeddings model. While this embeddings model does not need to be consistent with the LLM selection, it does need to be consistent between embedding our index and embedding our queries over that index.\n",
        "\n",
        "While we don't have to worry too much about that in this example - it's something to keep in mind for more complex applications.\n",
        "\n",
        "We're going to leverage a [`CacheBackedEmbeddings`](https://python.langchain.com/docs/modules/data_connection/caching_embeddings ) flow to prevent us from re-embedding similar queries over and over again.\n",
        "\n",
        "Not only will this save time, it will also save us precious embedding tokens, which will reduce the overall cost for our application.\n",
        "\n",
        ">#### Note:\n",
        ">The overall cost savings needs to be compared against the additional cost of storing the cached embeddings for a true cost/benefit analysis. If your users are submitting the same queries often, though, this pattern can be a massive reduction in cost.\n",
        "\n",
        "Documentation:\n",
        " - [`CacheBackedEmbeddings.from_bytes_store()`](https://api.python.langchain.com/en/latest/embeddings/langchain.embeddings.cache.CacheBackedEmbeddings.html#langchain.embeddings.cache.CacheBackedEmbeddings.from_bytes_store)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "AOzZWPU05WLr"
      },
      "outputs": [],
      "source": [
        "from langchain.embeddings.openai import OpenAIEmbeddings\n",
        "from langchain.embeddings import CacheBackedEmbeddings\n",
        "from langchain.storage import LocalFileStore\n",
        "\n",
        "store = LocalFileStore(\"./cache/\")\n",
        "\n",
        "core_embeddings_model = OpenAIEmbeddings()\n",
        "\n",
        "embedder = CacheBackedEmbeddings.from_bytes_store(\n",
        "    underlying_embeddings= core_embeddings_model,\n",
        "    document_embedding_cache= store,\n",
        "    namespace= \"dg\"\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that we have our `CacheBackedEmbeddings` pipeline set-up, let's index our documents into our Pinecone Vector Database. \n",
        "\n",
        "We'll add some useful metadata as well!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Review_Date</th>\n",
              "      <th>Author</th>\n",
              "      <th>Rating</th>\n",
              "      <th>Review_Title</th>\n",
              "      <th>Review</th>\n",
              "      <th>Review_Url</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>21 July 2023</td>\n",
              "      <td>LoveofLegacy</td>\n",
              "      <td>6</td>\n",
              "      <td>Beautiful film, but so preachy\\n</td>\n",
              "      <td>Margot does the best with what she's given, bu...</td>\n",
              "      <td>/review/rw9221648/?ref_=tt_urv</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Review_Date        Author Rating                       Review_Title  \\\n",
              "0  21 July 2023  LoveofLegacy      6   Beautiful film, but so preachy\\n   \n",
              "\n",
              "                                              Review  \\\n",
              "0  Margot does the best with what she's given, bu...   \n",
              "\n",
              "                       Review_Url  \n",
              "0  /review/rw9221648/?ref_=tt_urv  "
            ]
          },
          "execution_count": 24,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "data.head(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 26,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "26de409ed482438bb890d15875603266",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/100 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "from uuid import uuid4\n",
        "\n",
        "BATCH_LIMIT = 100\n",
        "\n",
        "texts = []\n",
        "metadatas = []\n",
        "\n",
        "for i in tqdm(range(len(data))):\n",
        "\n",
        "    record = data.iloc[i]\n",
        "\n",
        "    metadata = {\n",
        "        'review-url': str(record[\"Review_Url\"]),\n",
        "        'review-date' : str(record[\"Review_Date\"]),\n",
        "        'author' : str(record[\"Author\"]),\n",
        "        'rating' : str(record[\"Rating\"]),\n",
        "        'review-title' : str(record[\"Review_Title\"]),\n",
        "    }\n",
        "\n",
        "    record_texts = text_splitter.split_text(\n",
        "        text= str(record[\"Review\"]),\n",
        "        )\n",
        "\n",
        "    record_metadatas = [{\n",
        "        \"chunk\": j, \"text\": text, **metadata\n",
        "    } for j, text in enumerate(record_texts)]\n",
        "    texts.extend(record_texts)\n",
        "    metadatas.extend(record_metadatas)\n",
        "    \n",
        "    if len(texts) >= BATCH_LIMIT:\n",
        "        ids = [str(uuid4()) for _ in range(len(texts))]\n",
        "        embeds = embedder.embed_documents(texts)\n",
        "        index.upsert(vectors=zip(ids, embeds, metadatas))\n",
        "        texts = []\n",
        "        metadatas = []\n",
        "\n",
        "if len(texts) > 0:\n",
        "    ids = [str(uuid4()) for _ in range(len(texts))]\n",
        "    embeds = embedder.embed_documents(texts)\n",
        "    index.upsert(vectors=zip(ids, embeds, metadatas))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'dimension': 1536,\n",
              " 'index_fullness': 0.00117,\n",
              " 'namespaces': {'': {'vector_count': 117}},\n",
              " 'total_vector_count': 117}"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now that we've created our index, let's convert it to a LangChain `VectorStroe` so we can use it in the rest of the LangChain ecosystem!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {},
      "outputs": [],
      "source": [
        "from langchain.vectorstores import Pinecone\n",
        "\n",
        "text_field = \"text\"\n",
        "\n",
        "index = pinecone.Index(index_name)\n",
        "\n",
        "vectorstore = Pinecone(\n",
        "    index= index,\n",
        "    embedding= core_embeddings_model,\n",
        "    text_key=text_field,\n",
        "    ### YOUR CODE HERE, \n",
        "    ### YOUR CODE HERE, \n",
        "    ### YOUR CODE HERE\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IGHzcE5i6fOR"
      },
      "source": [
        "Now that we've created the VectorStore, we can check that it's working by embedding a query and retrieving passages from our reviews that are close to it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "[Document(page_content=\"The film's universe and settings are fantastic. The casting is really good too, with Gosling excelling in the role of Ken.\", metadata={'author': 'hamsterination', 'chunk': 0.0, 'rating': '6', 'review-date': datetime.datetime(2023, 7, 19, 0, 0), 'review-title': ' It could have been so much better...\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'}),\n",
              " Document(page_content='This movie is so much fun. It starts off really strong although the story does move away from \"Barbieland\" sooner than I would have liked. Nonetheless, it regains its footing with the final act in particular and I could not stop laughing at Ryan Gosling\\'s portrayal of Ken. That song will forever be stuck in my head.', metadata={'author': 'Genti25', 'chunk': 0.0, 'rating': '8', 'review-date': datetime.datetime(2023, 7, 20, 0, 0), 'review-title': ' You are Kenough\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'}),\n",
              " Document(page_content='I mean, Margo Robbie as Barbie just made perfect sense, but I really wanted to see the movie when I herd that Ryian Gosling was going to be Ken, I just thought that would be hilarious and I was right!', metadata={'author': 'subxerogravity', 'chunk': 0.0, 'rating': '8', 'review-date': datetime.datetime(2023, 7, 23, 0, 0), 'review-title': \" It's like G.I Joe...For girls! Seriously, and I love it for that!\\n\", 'review-url': '/review/rw9221648/?ref_=tt_urv'})]"
            ]
          },
          "execution_count": 32,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "query = \"Who played Ken?\"\n",
        "\n",
        "vectorstore.similarity_search(\n",
        "    query, \n",
        "    k=3  \n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ix4fyOUS-fU-"
      },
      "source": [
        "Let's see how much time the `CacheBackedEmbeddings` pattern saves us:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pCwtsJbc_KPd",
        "outputId": "92acfeb8-26e8-4dd8-d247-29fea2ca5187"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Retrying langchain.embeddings.openai.embed_with_retry.<locals>._embed_with_retry in 4.0 seconds as it raised RateLimitError: Rate limit reached for default-text-embedding-ada-002 in organization org-Im1pvmGA1xZ1sG3PHe3pOZja on requests per min. Limit: 3 / min. Please try again in 20s. Contact us through our help center at help.openai.com if you continue to have issues. Please add a payment method to your account to increase your rate limit. Visit https://platform.openai.com/account/billing to add a payment method..\n"
          ]
        },
        {
          "ename": "KeyboardInterrupt",
          "evalue": "",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[1;32m/Users/douggilmour/Documents/Training/llm/code/LLM-Ops-Cohort-2/Week 2/Tuesday - Retrieval Augmented Question Answering with LangChain/Barbie_Retrieval_Augmented_Question_Answering_(RAQA)_Assignment (Assignment).ipynb Cell 46\u001b[0m line \u001b[0;36m1\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/douggilmour/Documents/Training/llm/code/LLM-Ops-Cohort-2/Week%202/Tuesday%20-%20Retrieval%20Augmented%20Question%20Answering%20with%20LangChain/Barbie_Retrieval_Augmented_Question_Answering_%28RAQA%29_Assignment%20%28Assignment%29.ipynb#X63sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m get_ipython()\u001b[39m.\u001b[39;49mrun_cell_magic(\u001b[39m'\u001b[39;49m\u001b[39mtimeit\u001b[39;49m\u001b[39m'\u001b[39;49m, \u001b[39m'\u001b[39;49m\u001b[39m'\u001b[39;49m, \u001b[39m'\u001b[39;49m\u001b[39mquery = \u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39mI really wanted to enjoy this and I know that I am not the target audience but there were massive plot holes and no real flow.\u001b[39;49m\u001b[39m\"\u001b[39;49m\u001b[39m\\n\u001b[39;49;00m\u001b[39mvectorstore.similarity_search(\u001b[39;49m\u001b[39m\\n\u001b[39;49;00m\u001b[39m    query, \u001b[39;49m\u001b[39m\\n\u001b[39;49;00m\u001b[39m    k=3  \u001b[39;49m\u001b[39m\\n\u001b[39;49;00m\u001b[39m)\u001b[39;49m\u001b[39m\\n\u001b[39;49;00m\u001b[39m'\u001b[39;49m)\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2493\u001b[0m, in \u001b[0;36mInteractiveShell.run_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2491\u001b[0m \u001b[39mwith\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39mbuiltin_trap:\n\u001b[1;32m   2492\u001b[0m     args \u001b[39m=\u001b[39m (magic_arg_s, cell)\n\u001b[0;32m-> 2493\u001b[0m     result \u001b[39m=\u001b[39m fn(\u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n\u001b[1;32m   2495\u001b[0m \u001b[39m# The code below prevents the output from being displayed\u001b[39;00m\n\u001b[1;32m   2496\u001b[0m \u001b[39m# when using magics with decorator @output_can_be_silenced\u001b[39;00m\n\u001b[1;32m   2497\u001b[0m \u001b[39m# when the last Python token in the expression is a ';'.\u001b[39;00m\n\u001b[1;32m   2498\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39mgetattr\u001b[39m(fn, magic\u001b[39m.\u001b[39mMAGIC_OUTPUT_CAN_BE_SILENCED, \u001b[39mFalse\u001b[39;00m):\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/IPython/core/magics/execution.py:1189\u001b[0m, in \u001b[0;36mExecutionMagics.timeit\u001b[0;34m(self, line, cell, local_ns)\u001b[0m\n\u001b[1;32m   1186\u001b[0m         \u001b[39mif\u001b[39;00m time_number \u001b[39m>\u001b[39m\u001b[39m=\u001b[39m \u001b[39m0.2\u001b[39m:\n\u001b[1;32m   1187\u001b[0m             \u001b[39mbreak\u001b[39;00m\n\u001b[0;32m-> 1189\u001b[0m all_runs \u001b[39m=\u001b[39m timer\u001b[39m.\u001b[39;49mrepeat(repeat, number)\n\u001b[1;32m   1190\u001b[0m best \u001b[39m=\u001b[39m \u001b[39mmin\u001b[39m(all_runs) \u001b[39m/\u001b[39m number\n\u001b[1;32m   1191\u001b[0m worst \u001b[39m=\u001b[39m \u001b[39mmax\u001b[39m(all_runs) \u001b[39m/\u001b[39m number\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/timeit.py:206\u001b[0m, in \u001b[0;36mTimer.repeat\u001b[0;34m(self, repeat, number)\u001b[0m\n\u001b[1;32m    204\u001b[0m r \u001b[39m=\u001b[39m []\n\u001b[1;32m    205\u001b[0m \u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(repeat):\n\u001b[0;32m--> 206\u001b[0m     t \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtimeit(number)\n\u001b[1;32m    207\u001b[0m     r\u001b[39m.\u001b[39mappend(t)\n\u001b[1;32m    208\u001b[0m \u001b[39mreturn\u001b[39;00m r\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/IPython/core/magics/execution.py:173\u001b[0m, in \u001b[0;36mTimer.timeit\u001b[0;34m(self, number)\u001b[0m\n\u001b[1;32m    171\u001b[0m gc\u001b[39m.\u001b[39mdisable()\n\u001b[1;32m    172\u001b[0m \u001b[39mtry\u001b[39;00m:\n\u001b[0;32m--> 173\u001b[0m     timing \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49minner(it, \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mtimer)\n\u001b[1;32m    174\u001b[0m \u001b[39mfinally\u001b[39;00m:\n\u001b[1;32m    175\u001b[0m     \u001b[39mif\u001b[39;00m gcold:\n",
            "File \u001b[0;32m<magic-timeit>:2\u001b[0m, in \u001b[0;36minner\u001b[0;34m(_it, _timer)\u001b[0m\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/vectorstores/pinecone.py:226\u001b[0m, in \u001b[0;36mPinecone.similarity_search\u001b[0;34m(self, query, k, filter, namespace, **kwargs)\u001b[0m\n\u001b[1;32m    207\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39msimilarity_search\u001b[39m(\n\u001b[1;32m    208\u001b[0m     \u001b[39mself\u001b[39m,\n\u001b[1;32m    209\u001b[0m     query: \u001b[39mstr\u001b[39m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    213\u001b[0m     \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs: Any,\n\u001b[1;32m    214\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m List[Document]:\n\u001b[1;32m    215\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Return pinecone documents most similar to query.\u001b[39;00m\n\u001b[1;32m    216\u001b[0m \n\u001b[1;32m    217\u001b[0m \u001b[39m    Args:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    224\u001b[0m \u001b[39m        List of Documents most similar to the query and score for each\u001b[39;00m\n\u001b[1;32m    225\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 226\u001b[0m     docs_and_scores \u001b[39m=\u001b[39m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49msimilarity_search_with_score(\n\u001b[1;32m    227\u001b[0m         query, k\u001b[39m=\u001b[39;49mk, \u001b[39mfilter\u001b[39;49m\u001b[39m=\u001b[39;49m\u001b[39mfilter\u001b[39;49m, namespace\u001b[39m=\u001b[39;49mnamespace, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs\n\u001b[1;32m    228\u001b[0m     )\n\u001b[1;32m    229\u001b[0m     \u001b[39mreturn\u001b[39;00m [doc \u001b[39mfor\u001b[39;00m doc, _ \u001b[39min\u001b[39;00m docs_and_scores]\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/vectorstores/pinecone.py:172\u001b[0m, in \u001b[0;36mPinecone.similarity_search_with_score\u001b[0;34m(self, query, k, filter, namespace)\u001b[0m\n\u001b[1;32m    153\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39msimilarity_search_with_score\u001b[39m(\n\u001b[1;32m    154\u001b[0m     \u001b[39mself\u001b[39m,\n\u001b[1;32m    155\u001b[0m     query: \u001b[39mstr\u001b[39m,\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    158\u001b[0m     namespace: Optional[\u001b[39mstr\u001b[39m] \u001b[39m=\u001b[39m \u001b[39mNone\u001b[39;00m,\n\u001b[1;32m    159\u001b[0m ) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m List[Tuple[Document, \u001b[39mfloat\u001b[39m]]:\n\u001b[1;32m    160\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Return pinecone documents most similar to query, along with scores.\u001b[39;00m\n\u001b[1;32m    161\u001b[0m \n\u001b[1;32m    162\u001b[0m \u001b[39m    Args:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    169\u001b[0m \u001b[39m        List of Documents most similar to the query and score for each\u001b[39;00m\n\u001b[1;32m    170\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[1;32m    171\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39msimilarity_search_by_vector_with_score(\n\u001b[0;32m--> 172\u001b[0m         \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_embed_query(query), k\u001b[39m=\u001b[39mk, \u001b[39mfilter\u001b[39m\u001b[39m=\u001b[39m\u001b[39mfilter\u001b[39m, namespace\u001b[39m=\u001b[39mnamespace\n\u001b[1;32m    173\u001b[0m     )\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/vectorstores/pinecone.py:90\u001b[0m, in \u001b[0;36mPinecone._embed_query\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m     88\u001b[0m \u001b[39m\u001b[39m\u001b[39m\"\"\"Embed query text.\"\"\"\u001b[39;00m\n\u001b[1;32m     89\u001b[0m \u001b[39mif\u001b[39;00m \u001b[39misinstance\u001b[39m(\u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_embedding, Embeddings):\n\u001b[0;32m---> 90\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_embedding\u001b[39m.\u001b[39;49membed_query(text)\n\u001b[1;32m     91\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39m\u001b[39m.\u001b[39m_embedding(text)\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/embeddings/openai.py:518\u001b[0m, in \u001b[0;36mOpenAIEmbeddings.embed_query\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m    509\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39membed_query\u001b[39m(\u001b[39mself\u001b[39m, text: \u001b[39mstr\u001b[39m) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m List[\u001b[39mfloat\u001b[39m]:\n\u001b[1;32m    510\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Call out to OpenAI's embedding endpoint for embedding query text.\u001b[39;00m\n\u001b[1;32m    511\u001b[0m \n\u001b[1;32m    512\u001b[0m \u001b[39m    Args:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    516\u001b[0m \u001b[39m        Embedding for the text.\u001b[39;00m\n\u001b[1;32m    517\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 518\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49membed_documents([text])[\u001b[39m0\u001b[39m]\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/embeddings/openai.py:490\u001b[0m, in \u001b[0;36mOpenAIEmbeddings.embed_documents\u001b[0;34m(self, texts, chunk_size)\u001b[0m\n\u001b[1;32m    478\u001b[0m \u001b[39m\u001b[39m\u001b[39m\"\"\"Call out to OpenAI's embedding endpoint for embedding search docs.\u001b[39;00m\n\u001b[1;32m    479\u001b[0m \n\u001b[1;32m    480\u001b[0m \u001b[39mArgs:\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    486\u001b[0m \u001b[39m    List of embeddings, one for each text.\u001b[39;00m\n\u001b[1;32m    487\u001b[0m \u001b[39m\"\"\"\u001b[39;00m\n\u001b[1;32m    488\u001b[0m \u001b[39m# NOTE: to keep things simple, we assume the list may contain texts longer\u001b[39;00m\n\u001b[1;32m    489\u001b[0m \u001b[39m#       than the maximum context and use length-safe embedding function.\u001b[39;00m\n\u001b[0;32m--> 490\u001b[0m \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_get_len_safe_embeddings(texts, engine\u001b[39m=\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49mdeployment)\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/embeddings/openai.py:374\u001b[0m, in \u001b[0;36mOpenAIEmbeddings._get_len_safe_embeddings\u001b[0;34m(self, texts, engine, chunk_size)\u001b[0m\n\u001b[1;32m    371\u001b[0m     _iter \u001b[39m=\u001b[39m \u001b[39mrange\u001b[39m(\u001b[39m0\u001b[39m, \u001b[39mlen\u001b[39m(tokens), _chunk_size)\n\u001b[1;32m    373\u001b[0m \u001b[39mfor\u001b[39;00m i \u001b[39min\u001b[39;00m _iter:\n\u001b[0;32m--> 374\u001b[0m     response \u001b[39m=\u001b[39m embed_with_retry(\n\u001b[1;32m    375\u001b[0m         \u001b[39mself\u001b[39;49m,\n\u001b[1;32m    376\u001b[0m         \u001b[39minput\u001b[39;49m\u001b[39m=\u001b[39;49mtokens[i : i \u001b[39m+\u001b[39;49m _chunk_size],\n\u001b[1;32m    377\u001b[0m         \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49m\u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49m_invocation_params,\n\u001b[1;32m    378\u001b[0m     )\n\u001b[1;32m    379\u001b[0m     batched_embeddings\u001b[39m.\u001b[39mextend(r[\u001b[39m\"\u001b[39m\u001b[39membedding\u001b[39m\u001b[39m\"\u001b[39m] \u001b[39mfor\u001b[39;00m r \u001b[39min\u001b[39;00m response[\u001b[39m\"\u001b[39m\u001b[39mdata\u001b[39m\u001b[39m\"\u001b[39m])\n\u001b[1;32m    381\u001b[0m results: List[List[List[\u001b[39mfloat\u001b[39m]]] \u001b[39m=\u001b[39m [[] \u001b[39mfor\u001b[39;00m _ \u001b[39min\u001b[39;00m \u001b[39mrange\u001b[39m(\u001b[39mlen\u001b[39m(texts))]\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/langchain/embeddings/openai.py:107\u001b[0m, in \u001b[0;36membed_with_retry\u001b[0;34m(embeddings, **kwargs)\u001b[0m\n\u001b[1;32m    104\u001b[0m     response \u001b[39m=\u001b[39m embeddings\u001b[39m.\u001b[39mclient\u001b[39m.\u001b[39mcreate(\u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkwargs)\n\u001b[1;32m    105\u001b[0m     \u001b[39mreturn\u001b[39;00m _check_response(response, skip_empty\u001b[39m=\u001b[39membeddings\u001b[39m.\u001b[39mskip_empty)\n\u001b[0;32m--> 107\u001b[0m \u001b[39mreturn\u001b[39;00m _embed_with_retry(\u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkwargs)\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/tenacity/__init__.py:289\u001b[0m, in \u001b[0;36mBaseRetrying.wraps.<locals>.wrapped_f\u001b[0;34m(*args, **kw)\u001b[0m\n\u001b[1;32m    287\u001b[0m \u001b[39m@functools\u001b[39m\u001b[39m.\u001b[39mwraps(f)\n\u001b[1;32m    288\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39mwrapped_f\u001b[39m(\u001b[39m*\u001b[39margs: t\u001b[39m.\u001b[39mAny, \u001b[39m*\u001b[39m\u001b[39m*\u001b[39mkw: t\u001b[39m.\u001b[39mAny) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m t\u001b[39m.\u001b[39mAny:\n\u001b[0;32m--> 289\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39mself\u001b[39;49m(f, \u001b[39m*\u001b[39;49margs, \u001b[39m*\u001b[39;49m\u001b[39m*\u001b[39;49mkw)\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/tenacity/__init__.py:389\u001b[0m, in \u001b[0;36mRetrying.__call__\u001b[0;34m(self, fn, *args, **kwargs)\u001b[0m\n\u001b[1;32m    387\u001b[0m \u001b[39melif\u001b[39;00m \u001b[39misinstance\u001b[39m(do, DoSleep):\n\u001b[1;32m    388\u001b[0m     retry_state\u001b[39m.\u001b[39mprepare_for_next_attempt()\n\u001b[0;32m--> 389\u001b[0m     \u001b[39mself\u001b[39;49m\u001b[39m.\u001b[39;49msleep(do)\n\u001b[1;32m    390\u001b[0m \u001b[39melse\u001b[39;00m:\n\u001b[1;32m    391\u001b[0m     \u001b[39mreturn\u001b[39;00m do\n",
            "File \u001b[0;32m/opt/homebrew/Caskroom/miniforge/base/envs/llmops-day2/lib/python3.11/site-packages/tenacity/nap.py:31\u001b[0m, in \u001b[0;36msleep\u001b[0;34m(seconds)\u001b[0m\n\u001b[1;32m     25\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39msleep\u001b[39m(seconds: \u001b[39mfloat\u001b[39m) \u001b[39m-\u001b[39m\u001b[39m>\u001b[39m \u001b[39mNone\u001b[39;00m:\n\u001b[1;32m     26\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"\u001b[39;00m\n\u001b[1;32m     27\u001b[0m \u001b[39m    Sleep strategy that delays execution for a given number of seconds.\u001b[39;00m\n\u001b[1;32m     28\u001b[0m \n\u001b[1;32m     29\u001b[0m \u001b[39m    This is the default strategy, and may be mocked out for unit testing.\u001b[39;00m\n\u001b[1;32m     30\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[0;32m---> 31\u001b[0m     time\u001b[39m.\u001b[39msleep(seconds)\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ],
      "source": [
        "%%timeit\n",
        "query = \"I really wanted to enjoy this and I know that I am not the target audience but there were massive plot holes and no real flow.\"\n",
        "vectorstore.similarity_search(\n",
        "    query, \n",
        "    k=3  \n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4utL1EfARTm"
      },
      "source": [
        "As we can see, even over a significant number of runs - the cached query is significantly faster than the first instance of the query!\n",
        "\n",
        "With that, we're ready to move onto Task 3!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Po3kHNHGBp0j"
      },
      "source": [
        "### Task 3: Building a Retrieval Chain\n",
        "\n",
        "In this task, we'll be making a Retrieval Chain which will allow us to ask semantic questions over our data.\n",
        "\n",
        "This part is rather abstracted away from us in LangChain and so it seems very powerful.\n",
        "\n",
        "Be sure to check the documentation, the source code, and other provided resources to build a deeper understanding of what's happening \"under the hood\"!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i5Ux9XdzCDi9"
      },
      "source": [
        "#### A Basic RetrievalQA Chain\n",
        "\n",
        "We're going to leverage `return_source_documents=True` to ensure we have proper sources for our reviews - should the end user want to verify the reviews themselves.\n",
        "\n",
        "Hallucinations [are](https://arxiv.org/abs/2202.03629) [a](https://arxiv.org/abs/2305.15852) [massive](https://arxiv.org/abs/2303.16104) [problem](https://arxiv.org/abs/2305.18248) in LLM applications.\n",
        "\n",
        "Though it has been tenuously shown that using Retrieval Augmentation [reduces hallucination in conversations](https://arxiv.org/pdf/2104.07567.pdf), one sure fire way to ensure your model is not hallucinating in a non-transparent way is to provide sources with your responses. This way the end-user can verify the output."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O_waVo3AEk71"
      },
      "source": [
        "#### Our LLM\n",
        "\n",
        "In this notebook, we'll continue to leverage OpenAI's suite of models - this time we'll be using the `gpt-3.5-turbo` model to power our RetrievalQAWithSources chain."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "QfglvbBtExPR"
      },
      "outputs": [],
      "source": [
        "from langchain.llms.openai import OpenAIChat\n",
        "\n",
        "llm = OpenAIChat(\n",
        "    model=\"gpt-3.5-turbo\", \n",
        "    temperature=0.5,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w35ZkVEoE8II"
      },
      "source": [
        "Now we can set up our chain.\n",
        "\n",
        "First, we need to leverage our `VectorStore` as a retriever!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 38,
      "metadata": {
        "id": "T8TWJMH8F_w7"
      },
      "outputs": [],
      "source": [
        "retriever = vectorstore.as_retriever()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 39,
      "metadata": {
        "id": "QeD8R6huFIf6"
      },
      "outputs": [],
      "source": [
        "from langchain.chains import RetrievalQA\n",
        "from langchain.callbacks import StdOutCallbackHandler\n",
        "\n",
        "handler = StdOutCallbackHandler()\n",
        "\n",
        "qa_with_sources_chain = RetrievalQA.from_chain_type(\n",
        "    llm=llm,\n",
        "    retriever=retriever,\n",
        "    callbacks=[handler],\n",
        "    return_source_documents=True\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 40,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oqPaII9nF72R",
        "outputId": "0b46128f-edbe-4010-afdc-36603f25c4c6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Will Ferrell ruined every scene he was in.'"
            ]
          },
          "execution_count": 40,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "qa_with_sources_chain({\"query\" : \"How was Will Ferrell in this movie?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 41,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "47Ov7N22MxOS",
        "outputId": "aeb6271b-4161-4921-e07c-b5f9041e9c4d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Based on the given context, it is not possible to determine whether reviewers consider this movie \"Kenough\" or not.'"
            ]
          },
          "execution_count": 41,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "qa_with_sources_chain({\"query\" : \"Do reviewers consider this movie Kenough?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at the available metadata we have, thanks to our index-creation!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 42,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "result = qa_with_sources_chain({\"query\" : \"Was Will Ferrel funny?\"})"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 43,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Key: query\n",
            "Value: Was Will Ferrel funny?\n",
            "\n",
            "Key: result\n",
            "Value: Yes, the person stated that Will Ferrell ruined every scene he was in.\n",
            "\n",
            "Key: source_documents\n",
            "Value: [Document(page_content=\"I really wanted to enjoy this and I know that I am not the target audience but there were massive plot holes and no real flow. The film was very disjointed. Ryan Gosling as good as he is seemed to old to play Ken and Will Ferrell ruined every scene he was in. I just didn't get it, it seemed hollow artificial and hackneyed. A waste of some great talent. It was predictable without being reassuring and trying so hard to be woke in the most superficial way in that but trying to tick so many boxes it actually ticked none. Margo Robbie looks beautiful throughout, the costumes and the sets were amazing but the story was way too weak and didn't make much sense at all.\", metadata={'author': 'agjbull', 'chunk': 0.0, 'rating': '6', 'review-date': datetime.datetime(2023, 7, 23, 0, 0), 'review-title': ' Just a little empty\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'}), Document(page_content=\"The quality, the humor, and the writing of the movie is fun for a while. It's quirky and it's unique. When they get into the weeds and try to explore deeper themes, the movie is a miss. The middle expositional phase of the movie, I must say, is a bore.\", metadata={'author': 'Revuer223', 'chunk': 0.0, 'rating': '6', 'review-date': datetime.datetime(2023, 7, 23, 0, 0), 'review-title': ' Could Have Been Great. 2nd Half Brings It Down.\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'}), Document(page_content='While i\\'m not so sure at first, the movie kept getting even more fun, entertaining, and definitely better, also surprisingly deal with a legit serious stuff, Barbie is a weirdly fun movie that fills with this very interesting concept, definitely the first time that\\'s ever done, Greta Gerwig has created this whole new style of filmmaking specifically for Barbie, from the intentionally weird yet creative editing, some awkward and cringe scene, i found the comedy so funny instead of cringe, Barbie is one of the most original movie of the year and also one of the most original movie i\\'ve seen in a while, we all know Margot Robbie and Ryan Gosling is gonna carry the movie and they are, but Will Ferrell, Simu Liu, and the whole rest of the cast were also great and entertaining, the soundtrack was just great, except Nicki Minaj and Ice Spice \"Barbie World\" song that are just absolutely terrible, but Billie Eilish \"What Was I Made For?\" tune that kept haunting in the background until it', metadata={'author': 'HabibieHakim123', 'chunk': 1.0, 'rating': '8', 'review-date': datetime.datetime(2023, 7, 19, 0, 0), 'review-title': ' Barbie Is A Weirdly Fun Movie!\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'}), Document(page_content='I can imagine that in the boardroom they went through a discussion like this: People are going to think this is a feel good movie so we need to give them something deeper but also lets make it absurd and fast paced so that we can finish the plot quickly and also jump on the \"not taking itself serious\" bandwagon and also market these dolls again in the age of smartphone kids. The only reason this movie gets at 5 from me is the humor, which sometimes becomes very meta and creative and Margot Robbie and Ryan Gosling have acted very well. Besides this, they could have just given us a simple feel good movie and I would have respected the creators more. Pass.', metadata={'author': 'MaskedMinty', 'chunk': 2.0, 'rating': '5', 'review-date': datetime.datetime(2023, 7, 21, 0, 0), 'review-title': ' Preach Preach Preach\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'})]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for k, v in result.items():\n",
        "    print(f\"Key: {k}\")\n",
        "    print(f\"Value: {v}\")\n",
        "    print(\"\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 44,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Metadata: ('metadata', {'author': 'agjbull', 'chunk': 0.0, 'rating': '6', 'review-date': datetime.datetime(2023, 7, 23, 0, 0), 'review-title': ' Just a little empty\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'})\n",
            "Page Content: ('page_content', \"I really wanted to enjoy this and I know that I am not the target audience but there were massive plot holes and no real flow. The film was very disjointed. Ryan Gosling as good as he is seemed to old to play Ken and Will Ferrell ruined every scene he was in. I just didn't get it, it seemed hollow artificial and hackneyed. A waste of some great talent. It was predictable without being reassuring and trying so hard to be woke in the most superficial way in that but trying to tick so many boxes it actually ticked none. Margo Robbie looks beautiful throughout, the costumes and the sets were amazing but the story was way too weak and didn't make much sense at all.\")\n",
            "\n",
            "Metadata: ('metadata', {'author': 'Revuer223', 'chunk': 0.0, 'rating': '6', 'review-date': datetime.datetime(2023, 7, 23, 0, 0), 'review-title': ' Could Have Been Great. 2nd Half Brings It Down.\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'})\n",
            "Page Content: ('page_content', \"The quality, the humor, and the writing of the movie is fun for a while. It's quirky and it's unique. When they get into the weeds and try to explore deeper themes, the movie is a miss. The middle expositional phase of the movie, I must say, is a bore.\")\n",
            "\n",
            "Metadata: ('metadata', {'author': 'HabibieHakim123', 'chunk': 1.0, 'rating': '8', 'review-date': datetime.datetime(2023, 7, 19, 0, 0), 'review-title': ' Barbie Is A Weirdly Fun Movie!\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'})\n",
            "Page Content: ('page_content', 'While i\\'m not so sure at first, the movie kept getting even more fun, entertaining, and definitely better, also surprisingly deal with a legit serious stuff, Barbie is a weirdly fun movie that fills with this very interesting concept, definitely the first time that\\'s ever done, Greta Gerwig has created this whole new style of filmmaking specifically for Barbie, from the intentionally weird yet creative editing, some awkward and cringe scene, i found the comedy so funny instead of cringe, Barbie is one of the most original movie of the year and also one of the most original movie i\\'ve seen in a while, we all know Margot Robbie and Ryan Gosling is gonna carry the movie and they are, but Will Ferrell, Simu Liu, and the whole rest of the cast were also great and entertaining, the soundtrack was just great, except Nicki Minaj and Ice Spice \"Barbie World\" song that are just absolutely terrible, but Billie Eilish \"What Was I Made For?\" tune that kept haunting in the background until it')\n",
            "\n",
            "Metadata: ('metadata', {'author': 'MaskedMinty', 'chunk': 2.0, 'rating': '5', 'review-date': datetime.datetime(2023, 7, 21, 0, 0), 'review-title': ' Preach Preach Preach\\n', 'review-url': '/review/rw9221648/?ref_=tt_urv'})\n",
            "Page Content: ('page_content', 'I can imagine that in the boardroom they went through a discussion like this: People are going to think this is a feel good movie so we need to give them something deeper but also lets make it absurd and fast paced so that we can finish the plot quickly and also jump on the \"not taking itself serious\" bandwagon and also market these dolls again in the age of smartphone kids. The only reason this movie gets at 5 from me is the humor, which sometimes becomes very meta and creative and Margot Robbie and Ryan Gosling have acted very well. Besides this, they could have just given us a simple feel good movie and I would have respected the creators more. Pass.')\n",
            "\n"
          ]
        }
      ],
      "source": [
        "for page_content, metadata in result[\"source_documents\"]:\n",
        "    print(f\"Metadata: {metadata}\")\n",
        "    print(f\"Page Content: {page_content}\")\n",
        "    print(\"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Adding Prompt Caching and Monitoring\n",
        "\n",
        "Now that we have the basic `RetrievalQAChain` set up and working - let's add a few more tools to help us built a more performant application and add a visibility tool as well!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Visibility Tooling\n",
        "\n",
        "We'll be once again leveraging Weights and Biases as our visibility tool, so let's add that first!\n",
        "\n",
        "You'll want to use the same Weights and Biases account that you set-up last Thursday here!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 45,
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"WANDB_API_KEY\"] = getpass.getpass(\"Weights and Biases API Key:\")\n",
        "os.environ[\"WANDB_PROJECT\"] = \"barbie-retrieval-qa\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now, to set up WandB, all we have to do is..."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 46,
      "metadata": {},
      "outputs": [],
      "source": [
        "!pip install -q -U wandb"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 47,
      "metadata": {},
      "outputs": [],
      "source": [
        "os.environ[\"LANGCHAIN_WANDB_TRACING\"] = \"true\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Yes, that's it. \n",
        "\n",
        "Let's use our `RetrievalQA` chain to test it out!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 48,
      "metadata": {},
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Streaming LangChain activity to W&B at https://wandb.ai/dgilmour22/barbie-retrieval-qa/runs/li2t2wtl\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: `WandbTracer` is currently in beta.\n",
            "\u001b[34m\u001b[1mwandb\u001b[0m: Please report any issues to https://github.com/wandb/wandb/issues with the tag `langchain`.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Based on the given context, it is difficult to determine whether reviewers consider this movie \"Kenough\" or not.'"
            ]
          },
          "execution_count": 48,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "qa_with_sources_chain({\"query\" : \"Do reviewers consider this movie Kenough?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "With those simple lines of code - we've added full visibility to our prompts and responses through Weights and Biases! "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "#### Prompt Caching\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Adding A Prompt Cache\n",
        "\n",
        "The basic idea of Prompt Caching is to provide a way to circumvent going to the LLM for prompts we have already seen.\n",
        "\n",
        "Similar to cached embeddings, the idea is simple:\n",
        "\n",
        "- Keep track of all the input/output pairs\n",
        "- If a user query is (in the case of semantic similarity caches) close enough to a previous prompt contained in the cache, return the output associated with that pair"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Initializing a Prompt Cache\n",
        "\n",
        "There are many different tools you can use to implement a Prompt Cache - from a \"build it yourself\" VectorStore implementation - to Redis - to custom libraries - there are upsides and downsides to each solution. \n",
        "\n",
        "Let's look at the Redis-backed Cache vs. `InMemoryCache` as an example:\n",
        "\n",
        "Redis Cache\n",
        "| Pros  | Cons  |\n",
        "|---|---|\n",
        "| Managed and Robust  | Expensive to Host  |\n",
        "| Integrations on all Major Cloud Platforms  | Non-trivial to Integrate |\n",
        "| Easily Scalable  | Does not have a ChatModel implementation |\n",
        "\n",
        "`InMemoryCache`\n",
        "| Pros  | Cons  |\n",
        "|---|---|\n",
        "| Easily implemented  | Consumes potentially precious memory |\n",
        "| Completely Cloud Agnostic  | Does not offer inter-session caching |\n",
        "\n",
        "For the sake of ease of use - and to allow functionality with our `ChatOpenAI` model - we'll leverage `InMemoryCache`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "We need to set our `langchain.llm_cache` to use the `InMemoryCache`.\n",
        "\n",
        "- [`InMemoryCache`](https://api.python.langchain.com/en/latest/cache/langchain.cache.InMemoryCache.html)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 49,
      "metadata": {},
      "outputs": [],
      "source": [
        "import langchain\n",
        "from langchain.cache import InMemoryCache\n",
        "langchain.llm_cache = InMemoryCache()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "One more important fact about the `InMemoryCache` is that it is what's called an \"exact-match\" cache - meaning it will only trigger when the user query is *exactly* represented in the cache. \n",
        "\n",
        "This is a safer cache, as we can guarentee the user's query exactly matches with previous queries and we don't have to worry about edge-cases where semantic similarity might fail - but it does reduce the potential to hit the cache.\n",
        "\n",
        "We could leverage tools like `GPTCache`, or `RedisCache` (for non-chat model implementations) to get a \"semantic similarity\" cache, if desired!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 50,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "CPU times: user 13.1 ms, sys: 2.6 ms, total: 15.7 ms\n",
            "Wall time: 1.79 s\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Based on the given context, it is unclear whether reviewers consider this movie \"Kenough\" or not.'"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%time\n",
        "qa_with_sources_chain({\"query\" : \"Do reviewers consider this movie Kenough?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 51,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "CPU times: user 20.1 ms, sys: 3.02 ms, total: 23.1 ms\n",
            "Wall time: 385 ms\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Based on the given context, it is unclear whether reviewers consider this movie \"Kenough\" or not.'"
            ]
          },
          "execution_count": 51,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%time\n",
        "qa_with_sources_chain({\"query\" : \"Do reviewers consider this movie Kenough?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Let's look at an example that is extremely close - but is not the exact query."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 52,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "\n",
            "\u001b[1m> Entering new RetrievalQA chain...\u001b[0m\n",
            "\n",
            "\u001b[1m> Finished chain.\u001b[0m\n",
            "CPU times: user 12.8 ms, sys: 2.15 ms, total: 14.9 ms\n",
            "Wall time: 1.77 s\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'Based on the given context, it is unclear whether reviewers consider the movie \"Kenough\" or not.'"
            ]
          },
          "execution_count": 52,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "%%time\n",
        "qa_with_sources_chain({\"query\" : \"Do reviewers consider this here movie Kenough?\"})[\"result\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As you can see, adding an exact-match prompt cache is a very small lift - but it can significantly improve the latency of your end-user application experience!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QQVCkoy3H5Rw"
      },
      "source": [
        "### Conclusion\n",
        "\n",
        "And with that, we have our Barbie Review RAQA Application built!\n",
        "\n",
        "Let's port it into a Chainlit app and put it up on a Hugging Face Space!"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyOiWOWtcIc3OpL6fUrMAdOt",
      "include_colab_link": true,
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}