This is a personal proof-of-concept project that uses a Retrieval-Augmented Generation (RAG) Large Language Model (LLM) implementation to derive insights from Snapchat user reviews. Specifically, this uses Claude models (2 and Sonnet) via Amazon Bedrock, Amazon Titan Embeddings, and Meta's Facebook AI Similarity Search (FAISS) vector store to classify user reviews (i.e., positive, neutral, or negative), extract named products referenced for product tagging, and succinctly summarizing each review.
Disclaimer: This work is in a personal capacity unrelated to my employer. It relies only on publicly available resources. This package is for illustrative purposes and is not designed for end-to-end productionization as-is. Also consider that this is an extremely small sample size of user reviews and should therefore not be perceived as representative of the average user experience. For example, the reviews publicly available per the methodology below are apparently mostly negative, whereas the Apple App Store rating of 4.6/5.0 demonstrates that most reviews are positive.
You will need your own Amazon Web Services (AWS) account with Claude and Titan Amazon Bedrock model access. Your Python environment will also require:
- langchain>=0.1.11
- langchain-community
- faiss-cpu==1.8.0
This package will demonstrate how to:
- Import libraries
- Instantiate the LLM and embeddings models
- Load 10 Snapchat reviews as documents
- Split documents into chunks
- Confirm embeddings functionality
- Create vector store
- Embed question and return relevant chunks
- Create RAG prompt template
- Produce RAG outputs with Claude 2
- Define Claude 3 function
- Read in reviews as .csv for iterative prompting
- Define function to prompt reviews
- Generate review sentiments, identify products mentioned, and generate summaries
When Claude 2 was provided with the vector store of user reviews and prompted "What are some of the features that people like most about Snapchat?", it returned:
When Claude 2 was provided with the vector store of user reviews and prompted "What are the most important issues users are experiencing with Snapchat?", it returned:
When Claude 3 Sonnet was iteratively prompted to classify the sentiment of the summary, extract any named products, and summarize the three most important words in five words or fewer each, it returned (sample of five results):
The Snapchat user reviews used in this project are the 10 publicly-available Apple App Store user reviews found here. Although usernames are present at the url provided, these were masked in this application to ensure user privacy. While a larger dataset would be more valuable for generating insights at scale, large Kaggle datasets of Snapchat user reviews were not used, explicitly to avoid any perceived infringement using user data of dubious provenance. Additionally, while changing the region in the above URL (e.g., to 'gb' instead of 'us') would have yielded additional user reviews, those reviews were avoided to ensure regulatory compliance (i.e., GDPR, etc.).
Future next steps for this project include:
- Incorporating evaluation methodologies to assess the quality of the outputs beyond the current heuristic assessment
- Inspecting the methodological decisions with further granularity (e.g., the chunk size during the chunking process, etc.)
- Applying this approach to additional use cases
This project is licensed under the MIT License