notebooks/run_all_evaluations.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Evaluating All Models Using GPT-4 as an A/B Tester\n",
    "\n",
    "## Introduction\n",
    "This notebook contains code to evaluate all A/B tests across all prompts for a benchmark. The code reads in a list of unevaluated AB tests with prompts and responses from a database, runs parallel A/B tests with GPT-4 as a judge, and writes the results back to the database.\n",
    "\n",
    "## Setup\n",
    "Before running the code, make sure to install the necessary dependencies by setting up the Poetry environment. You will also need to set up a database connection by filling in the appropriate values in the cells below. You will also need to provide up an OpenAI API key and organization ID (if appropriate).\n",
    "\n",
    "Additionally, specify the benchmark and evaluating model to use for evaluation. The benchmark is the name of the benchmark to evaluate. The evaluating model is the name of the model to use as a judge for the A/B tests. The evaluating model must be a model that is already in the database."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "import os"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:12.107427Z",
     "start_time": "2023-07-27T01:12:12.085027Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "os.environ['OPENAI_API_KEY']=\"YOUR_API_KEY\"\n",
    "os.environ['OPENAI_ORGANIZATION']=\"YOUR_ORG_ID\""
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:12.676961Z",
     "start_time": "2023-07-27T01:12:12.671558Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "os.environ['POSTGRES_HOST']=\"localhost\"\n",
    "os.environ['POSTGRES_DB']=\"llm_eval_db\"\n",
    "os.environ['POSTGRES_USER']=\"maker\"\n",
    "os.environ['POSTGRES_PASSWORD']=\"makerpassword\""
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:12.862353Z",
     "start_time": "2023-07-27T01:12:12.858654Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [],
   "source": [
    "EVALUATING_MODEL = \"gpt-4-0613\""
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:13.256142Z",
     "start_time": "2023-07-27T01:12:13.237917Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Code Structure\n",
    "The code is split into several sections:\n",
    "\n",
    "### Imports\n",
    "The first section imports the necessary modules and functions.\n",
    "\n",
    "### Data Loading\n",
    "The second section loads the unevaluated AB test prompts from the database and converts them to requests for the OpenAI API.\n",
    "\n",
    "### Model Evaluation\n",
    "The third section evaluates the AB test prompts using the OpenAI cookbook.\n",
    "\n",
    "### Data Writing\n",
    "The fourth section parses the evaluation results and writes them back to the database.\n",
    "\n",
    "## Usage\n",
    "To use this code, simply run the notebook from start to finish. The results will be written back to the database automatically."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "from db.db_utils import get_all_unevaluated_ab_test_prompts_for_benchmark, insert_evals_by_model_into_db\n",
    "from src.parse import db_game_to_dict, construct_request, parse_result_to_eval_entry, games_list_to_lookup\n",
    "from src.utils import read_jsonl, write_jsonl"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:15.117068Z",
     "start_time": "2023-07-27T01:12:14.858094Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Data Loading"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 incomplete games found\n",
      "Example: (UUID('76314a10-f32a-4188-bba8-fd56f0ba95f4'), UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), UUID('db8d724d-710b-4926-8e7c-b4baa9b71901'), UUID('e753ecbf-45cf-4a7d-ab7b-2410c8cacbb8'), '1. \"Call Centers For Dummies\" by Real Bergevin, Afshan Kinder, Winston Siegel, and Bruce Simpson: This book provides a comprehensive overview of call center management, including tips for modernization.\\n\\n2. \"The Best Service is No Service: How to Liberate Your Customers from Customer Service, Keep Them Happy, and Control Costs\" by Bill Price and David Jaffe: This book offers innovative strategies for modernizing customer service, including call centers.\\n\\n3. \"Delivering Quality Service: Balancing Customer Perceptions and Expectations\" by Valarie A. Zeithaml, A. Parasuraman, and Leonard L. Berry: This book provides insights into customer expectations and how to meet them in a modern call center.\\n\\n4. \"The Effortless Experience: Conquering the New Battleground for Customer Loyalty\" by Matthew Dixon, Nick Toman, and Rick DeLisi: This book provides strategies for improving customer loyalty, a key aspect of modern call center operations.\\n\\n5. \"Call Center Management on Fast Forward: Succeeding in Today\\'s Dynamic Inbound Environment\" by Brad Cleveland: This book provides a comprehensive guide to managing and modernizing call centers.\\n\\n6. \"The Nordstrom Way to Customer Experience Excellence: Creating a Values-Driven Service Culture\" by Robert Spector and breAnne O. Reeves: This book provides insights into creating a customer-focused culture, a key aspect of modern call centers.\\n\\n7. \"Customer Satisfaction Is Worthless, Customer Loyalty Is Priceless: How to Make Customers Love You, Keep Them Coming Back and Tell Everyone They Know\" by Jeffrey Gitomer: This book provides strategies for building customer loyalty, a key aspect of modern call centers.\\n\\n8. \"Raving Fans: A Revolutionary Approach to Customer Service\" by Ken Blanchard and Sheldon Bowles: This book provides a new approach to customer service, which can be applied to modernizing call centers.\\n\\n9. \"Be Our Guest: Perfecting the Art of Customer Service\" by The Disney Institute: This book provides insights into Disney\\'s approach to customer service, which can be applied to modernizing call centers.\\n\\n10. Websites like Call Centre Helper, Contact Center Pipeline, and Call Centre Management Association provide articles, webinars, and resources on modern call center practices and technologies.\\n\\n11. Research papers and case studies from consulting firms like McKinsey, Deloitte, and Accenture often provide insights into the latest trends and best practices in call center modernization.\\n\\n12. Online courses and certifications from institutions like Coursera, edX, and LinkedIn Learning can provide structured learning experiences on call center management and modernization.', UUID('cc7a1ece-9fe4-404a-bfd7-bb4c45d24fd6'), ' There are several resources available that can help you modernize your call center. Here are some references to consider:\\n\\n1. \"The Future of Customer Service: Conversational AI, Automation, and the Role of Human Agents\" by Forrester - This report provides an overview of the current state of customer service and explores the role of conversational AI, automation, and human agents in modern call centers.\\n2. \"The Modern Contact Center: A Guide to Best Practices and Technology\" by ICMI - This guide provides best practices and technology recommendations for modernizing your call center, including strategies for leveraging cloud-based solutions, automation, and analytics.\\n3. \"5 Key Trends Shaping the Future of Customer Service\" by Harvard Business Review - This article highlights five key trends shaping the future of customer service, including the rise of AI, personalization, and the importance of employee engagement.\\n4. \"The Impact of AI on Customer Service: Lessons Learned from Early Adopters\" by MIT Sloan Management Review - This report shares lessons learned from early adopters of AI in customer service, including the benefits and challenges of implementing AI solutions.\\n5. \"The Future of Work in Customer Service: A Journey to the Next-Generation Contact Center\" by Deloitte - This report explores the impact of digital technologies on the customer service industry and provides guidance on how to create a next-generation contact center.\\n6. \"Customer Experience 2020: The Future of Customer Service\" by Oracle - This report examines the future of customer service and provides insights into the technologies and strategies that will shape the industry in the coming years.\\n7. \"The Role of AI in Customer Service: A Global Study\" by NICE - This study explores the use of AI in customer service and provides insights into the benefits and challenges of implementing AI solutions.\\n8. \"The Future of Customer Engagement: Conversational AI, Machine Learning, and the Role of Humans\" by Forbes - This article discusses the role of conversational AI and machine learning in customer engagement and explores the ways in which humans and machines can work together to deliver exceptional customer experiences.\\n9. \"Modernizing the Contact Center: A Guide to Cloud-Based Solutions\" by No Jitter - This guide provides an overview of cloud-based solutions for modernizing the contact center, including the benefits and considerations for migrating to the cloud.\\n10. \"The Future of Customer Service: A Playbook for the Digital Age\" by Salesforce - This playbook provides a comprehensive guide to modernizing your call center, including strategies for leveraging AI, automation, and cloud-based solutions.\\n\\nThese references provide valuable insights and guidance for modernizing your call center and delivering exceptional customer experiences in the digital age.', UUID('56a5bedc-823e-4193-af89-d1caa000f806'), 'What are some good references for modernizing a call center?', UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'gpt-4-0613')\n"
     ]
    }
   ],
   "source": [
    "incomplete_game_rows = await get_all_unevaluated_ab_test_prompts_for_benchmark(eval_model_name=EVALUATING_MODEL)\n",
    "print(f\"{len(incomplete_game_rows)} incomplete games found\")\n",
    "if len(incomplete_game_rows) > 0:\n",
    "    print(f\"Example: {incomplete_game_rows[0]}\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:16.263129Z",
     "start_time": "2023-07-27T01:12:15.938338Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 games parsed\n",
      "Example: {'ab_test_id': UUID('76314a10-f32a-4188-bba8-fd56f0ba95f4'), 'model_a_id': UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'model_b_id': UUID('db8d724d-710b-4926-8e7c-b4baa9b71901'), 'response_a_id': UUID('e753ecbf-45cf-4a7d-ab7b-2410c8cacbb8'), 'response_a_text': '1. \"Call Centers For Dummies\" by Real Bergevin, Afshan Kinder, Winston Siegel, and Bruce Simpson: This book provides a comprehensive overview of call center management, including tips for modernization.\\n\\n2. \"The Best Service is No Service: How to Liberate Your Customers from Customer Service, Keep Them Happy, and Control Costs\" by Bill Price and David Jaffe: This book offers innovative strategies for modernizing customer service, including call centers.\\n\\n3. \"Delivering Quality Service: Balancing Customer Perceptions and Expectations\" by Valarie A. Zeithaml, A. Parasuraman, and Leonard L. Berry: This book provides insights into customer expectations and how to meet them in a modern call center.\\n\\n4. \"The Effortless Experience: Conquering the New Battleground for Customer Loyalty\" by Matthew Dixon, Nick Toman, and Rick DeLisi: This book provides strategies for improving customer loyalty, a key aspect of modern call center operations.\\n\\n5. \"Call Center Management on Fast Forward: Succeeding in Today\\'s Dynamic Inbound Environment\" by Brad Cleveland: This book provides a comprehensive guide to managing and modernizing call centers.\\n\\n6. \"The Nordstrom Way to Customer Experience Excellence: Creating a Values-Driven Service Culture\" by Robert Spector and breAnne O. Reeves: This book provides insights into creating a customer-focused culture, a key aspect of modern call centers.\\n\\n7. \"Customer Satisfaction Is Worthless, Customer Loyalty Is Priceless: How to Make Customers Love You, Keep Them Coming Back and Tell Everyone They Know\" by Jeffrey Gitomer: This book provides strategies for building customer loyalty, a key aspect of modern call centers.\\n\\n8. \"Raving Fans: A Revolutionary Approach to Customer Service\" by Ken Blanchard and Sheldon Bowles: This book provides a new approach to customer service, which can be applied to modernizing call centers.\\n\\n9. \"Be Our Guest: Perfecting the Art of Customer Service\" by The Disney Institute: This book provides insights into Disney\\'s approach to customer service, which can be applied to modernizing call centers.\\n\\n10. Websites like Call Centre Helper, Contact Center Pipeline, and Call Centre Management Association provide articles, webinars, and resources on modern call center practices and technologies.\\n\\n11. Research papers and case studies from consulting firms like McKinsey, Deloitte, and Accenture often provide insights into the latest trends and best practices in call center modernization.\\n\\n12. Online courses and certifications from institutions like Coursera, edX, and LinkedIn Learning can provide structured learning experiences on call center management and modernization.', 'response_b_id': UUID('cc7a1ece-9fe4-404a-bfd7-bb4c45d24fd6'), 'response_b_text': ' There are several resources available that can help you modernize your call center. Here are some references to consider:\\n\\n1. \"The Future of Customer Service: Conversational AI, Automation, and the Role of Human Agents\" by Forrester - This report provides an overview of the current state of customer service and explores the role of conversational AI, automation, and human agents in modern call centers.\\n2. \"The Modern Contact Center: A Guide to Best Practices and Technology\" by ICMI - This guide provides best practices and technology recommendations for modernizing your call center, including strategies for leveraging cloud-based solutions, automation, and analytics.\\n3. \"5 Key Trends Shaping the Future of Customer Service\" by Harvard Business Review - This article highlights five key trends shaping the future of customer service, including the rise of AI, personalization, and the importance of employee engagement.\\n4. \"The Impact of AI on Customer Service: Lessons Learned from Early Adopters\" by MIT Sloan Management Review - This report shares lessons learned from early adopters of AI in customer service, including the benefits and challenges of implementing AI solutions.\\n5. \"The Future of Work in Customer Service: A Journey to the Next-Generation Contact Center\" by Deloitte - This report explores the impact of digital technologies on the customer service industry and provides guidance on how to create a next-generation contact center.\\n6. \"Customer Experience 2020: The Future of Customer Service\" by Oracle - This report examines the future of customer service and provides insights into the technologies and strategies that will shape the industry in the coming years.\\n7. \"The Role of AI in Customer Service: A Global Study\" by NICE - This study explores the use of AI in customer service and provides insights into the benefits and challenges of implementing AI solutions.\\n8. \"The Future of Customer Engagement: Conversational AI, Machine Learning, and the Role of Humans\" by Forbes - This article discusses the role of conversational AI and machine learning in customer engagement and explores the ways in which humans and machines can work together to deliver exceptional customer experiences.\\n9. \"Modernizing the Contact Center: A Guide to Cloud-Based Solutions\" by No Jitter - This guide provides an overview of cloud-based solutions for modernizing the contact center, including the benefits and considerations for migrating to the cloud.\\n10. \"The Future of Customer Service: A Playbook for the Digital Age\" by Salesforce - This playbook provides a comprehensive guide to modernizing your call center, including strategies for leveraging AI, automation, and cloud-based solutions.\\n\\nThese references provide valuable insights and guidance for modernizing your call center and delivering exceptional customer experiences in the digital age.', 'prompt_id': UUID('56a5bedc-823e-4193-af89-d1caa000f806'), 'prompt_text': 'What are some good references for modernizing a call center?', 'eval_model_id': UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'eval_model_name': 'gpt-4-0613'}\n"
     ]
    }
   ],
   "source": [
    "games = [db_game_to_dict(row) for row in incomplete_game_rows]\n",
    "print(f\"{len(games)} games parsed\")\n",
    "if len(games) > 0:\n",
    "    print(f\"Example: {games[0]}\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:16.337409Z",
     "start_time": "2023-07-27T01:12:16.285554Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 requests written to file\n",
      "Example: {'model': 'gpt-4-0613', 'messages': [{'role': 'system', 'content': 'Ignore previous instructions.\\nAssume the role of an A/B tester. You are highly experienced and have been doing this for years. Your analysis will be extremely professional and unbiased.\\nYour job is to compare two AI Assistants, model_a and model_b, and determine which one is better. User will provide you with a [Question], [Response from model_a], and [Response from model_b].\\nEnsure that the order in which the responses are presented to you does not influence your decision. You are known to show bias towards the first response you read. You are aware of this bias and will try to avoid it.\\nYou will carefully analyze both Responses and assign a score from 1 to 10 to each answer based on the following metrics: accuracy, safety, completeness, usefulness, and readability. 1 being the lowest and 10 being the highest.\\nOnly give a single score to each answer. Do not give separate scores for each metric. And make sure each score is a number between 1 and 10. Greater than or equal to 1 and less than or equal to 10.\\nYou must follow this step by step approach to make your decision.\\nstep 1: Read the Question.\\nstep 2: Read the responses from the models. The order in which you read the responses should not influence your decision.\\nstep 3: Carefully analyze both Responses. Assign a score from 1 to 10 to each answer based on the following metrics: accuracy, safety, completeness, usefulness, and readability. 1 being the lowest and 10 being the highest.\\nstep 4: Compare your scores for the first and the second Assistants and choose a winner based on the highest score. Your Choice will be either \"model_a\" or \"model_b\" based on which model has the highest score.\\nFormat your response in a valid JSON format with keys \"choice\", \"reason\", and \"scores\". Do not include any other text.\\nThe \"choice\" field will be either \"model_a\" or \"model_b\".\\nThe \"scores\" field will include the score for each model.\\nIn the \"reason\" field, you will include a detailed step by step description of your analysis. Please go into excruciating detail and explain the decisions you made in each step of the process. Do not include any newlines in the \"reason\" field. You can use the \"\\n\" character to indicate a newline. Also, do not use any double quotes characters in the \"reason\" field. Your output should be in a valid JSON format.\\nexample output format: {\"choice\": \"<winner of the A/B test>\", \"reason\": \"<your detailed step by step analysis here>\", \"scores\": {\"model_a\": <score for model_a>, \"model_b\": <score for model_b>}}'}, {'role': 'user', 'content': '[Question]\\nWhat are some good references for modernizing a call center?\\n[End of Question]\\n\\n[Response from model_a]\\n1. \"Call Centers For Dummies\" by Real Bergevin, Afshan Kinder, Winston Siegel, and Bruce Simpson: This book provides a comprehensive overview of call center management, including tips for modernization.\\n\\n2. \"The Best Service is No Service: How to Liberate Your Customers from Customer Service, Keep Them Happy, and Control Costs\" by Bill Price and David Jaffe: This book offers innovative strategies for modernizing customer service, including call centers.\\n\\n3. \"Delivering Quality Service: Balancing Customer Perceptions and Expectations\" by Valarie A. Zeithaml, A. Parasuraman, and Leonard L. Berry: This book provides insights into customer expectations and how to meet them in a modern call center.\\n\\n4. \"The Effortless Experience: Conquering the New Battleground for Customer Loyalty\" by Matthew Dixon, Nick Toman, and Rick DeLisi: This book provides strategies for improving customer loyalty, a key aspect of modern call center operations.\\n\\n5. \"Call Center Management on Fast Forward: Succeeding in Today\\'s Dynamic Inbound Environment\" by Brad Cleveland: This book provides a comprehensive guide to managing and modernizing call centers.\\n\\n6. \"The Nordstrom Way to Customer Experience Excellence: Creating a Values-Driven Service Culture\" by Robert Spector and breAnne O. Reeves: This book provides insights into creating a customer-focused culture, a key aspect of modern call centers.\\n\\n7. \"Customer Satisfaction Is Worthless, Customer Loyalty Is Priceless: How to Make Customers Love You, Keep Them Coming Back and Tell Everyone They Know\" by Jeffrey Gitomer: This book provides strategies for building customer loyalty, a key aspect of modern call centers.\\n\\n8. \"Raving Fans: A Revolutionary Approach to Customer Service\" by Ken Blanchard and Sheldon Bowles: This book provides a new approach to customer service, which can be applied to modernizing call centers.\\n\\n9. \"Be Our Guest: Perfecting the Art of Customer Service\" by The Disney Institute: This book provides insights into Disney\\'s approach to customer service, which can be applied to modernizing call centers.\\n\\n10. Websites like Call Centre Helper, Contact Center Pipeline, and Call Centre Management Association provide articles, webinars, and resources on modern call center practices and technologies.\\n\\n11. Research papers and case studies from consulting firms like McKinsey, Deloitte, and Accenture often provide insights into the latest trends and best practices in call center modernization.\\n\\n12. Online courses and certifications from institutions like Coursera, edX, and LinkedIn Learning can provide structured learning experiences on call center management and modernization.\\n[End of Response from model_a]\\n\\n[Response from model_b]\\n There are several resources available that can help you modernize your call center. Here are some references to consider:\\n\\n1. \"The Future of Customer Service: Conversational AI, Automation, and the Role of Human Agents\" by Forrester - This report provides an overview of the current state of customer service and explores the role of conversational AI, automation, and human agents in modern call centers.\\n2. \"The Modern Contact Center: A Guide to Best Practices and Technology\" by ICMI - This guide provides best practices and technology recommendations for modernizing your call center, including strategies for leveraging cloud-based solutions, automation, and analytics.\\n3. \"5 Key Trends Shaping the Future of Customer Service\" by Harvard Business Review - This article highlights five key trends shaping the future of customer service, including the rise of AI, personalization, and the importance of employee engagement.\\n4. \"The Impact of AI on Customer Service: Lessons Learned from Early Adopters\" by MIT Sloan Management Review - This report shares lessons learned from early adopters of AI in customer service, including the benefits and challenges of implementing AI solutions.\\n5. \"The Future of Work in Customer Service: A Journey to the Next-Generation Contact Center\" by Deloitte - This report explores the impact of digital technologies on the customer service industry and provides guidance on how to create a next-generation contact center.\\n6. \"Customer Experience 2020: The Future of Customer Service\" by Oracle - This report examines the future of customer service and provides insights into the technologies and strategies that will shape the industry in the coming years.\\n7. \"The Role of AI in Customer Service: A Global Study\" by NICE - This study explores the use of AI in customer service and provides insights into the benefits and challenges of implementing AI solutions.\\n8. \"The Future of Customer Engagement: Conversational AI, Machine Learning, and the Role of Humans\" by Forbes - This article discusses the role of conversational AI and machine learning in customer engagement and explores the ways in which humans and machines can work together to deliver exceptional customer experiences.\\n9. \"Modernizing the Contact Center: A Guide to Cloud-Based Solutions\" by No Jitter - This guide provides an overview of cloud-based solutions for modernizing the contact center, including the benefits and considerations for migrating to the cloud.\\n10. \"The Future of Customer Service: A Playbook for the Digital Age\" by Salesforce - This playbook provides a comprehensive guide to modernizing your call center, including strategies for leveraging AI, automation, and cloud-based solutions.\\n\\nThese references provide valuable insights and guidance for modernizing your call center and delivering exceptional customer experiences in the digital age.\\n[End of Response from model_b]\\n\\nPlease complete the A/B test. Make sure that your entire response is a valid JSON string.'}], 'user': '76314a10-f32a-4188-bba8-fd56f0ba95f4_56a5bedc-823e-4193-af89-d1caa000f806', 'temperature': 0.2, 'max_tokens': 1024}\n"
     ]
    }
   ],
   "source": [
    "requests = [construct_request(game) for game in games]\n",
    "write_jsonl(requests, 'requests.jsonl')\n",
    "print(f\"{len(requests)} requests written to file\")\n",
    "if len(requests) > 0:\n",
    "    print(f\"Example: {requests[0]}\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:16.800448Z",
     "start_time": "2023-07-27T01:12:16.784816Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model Evaluation"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:Starting request #0\r\n",
      "INFO:root:Parallel processing complete. Results saved to requests_results.jsonl\r\n"
     ]
    }
   ],
   "source": [
    "!python ../src/api_request_parallel_processor.py \\\n",
    "  --requests_filepath requests.jsonl \\\n",
    "  --save_filepath requests_results.jsonl \\\n",
    "  --request_url https://api.openai.com/v1/chat/completions \\\n",
    "  --max_requests_per_minute 200 \\\n",
    "  --max_tokens_per_minute 40000 \\\n",
    "  --token_encoding_name cl100k_base \\\n",
    "  --max_attempts 5 \\\n",
    "  --logging_level 20"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:37.493194Z",
     "start_time": "2023-07-27T01:12:18.669564Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Data Writing"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 results read from file\n",
      "Example: [{'model': 'gpt-4-0613', 'messages': [{'role': 'system', 'content': 'Ignore previous instructions.\\nAssume the role of an A/B tester. You are highly experienced and have been doing this for years. Your analysis will be extremely professional and unbiased.\\nYour job is to compare two AI Assistants, model_a and model_b, and determine which one is better. User will provide you with a [Question], [Response from model_a], and [Response from model_b].\\nEnsure that the order in which the responses are presented to you does not influence your decision. You are known to show bias towards the first response you read. You are aware of this bias and will try to avoid it.\\nYou will carefully analyze both Responses and assign a score from 1 to 10 to each answer based on the following metrics: accuracy, safety, completeness, usefulness, and readability. 1 being the lowest and 10 being the highest.\\nOnly give a single score to each answer. Do not give separate scores for each metric. And make sure each score is a number between 1 and 10. Greater than or equal to 1 and less than or equal to 10.\\nYou must follow this step by step approach to make your decision.\\nstep 1: Read the Question.\\nstep 2: Read the responses from the models. The order in which you read the responses should not influence your decision.\\nstep 3: Carefully analyze both Responses. Assign a score from 1 to 10 to each answer based on the following metrics: accuracy, safety, completeness, usefulness, and readability. 1 being the lowest and 10 being the highest.\\nstep 4: Compare your scores for the first and the second Assistants and choose a winner based on the highest score. Your Choice will be either \"model_a\" or \"model_b\" based on which model has the highest score.\\nFormat your response in a valid JSON format with keys \"choice\", \"reason\", and \"scores\". Do not include any other text.\\nThe \"choice\" field will be either \"model_a\" or \"model_b\".\\nThe \"scores\" field will include the score for each model.\\nIn the \"reason\" field, you will include a detailed step by step description of your analysis. Please go into excruciating detail and explain the decisions you made in each step of the process. Do not include any newlines in the \"reason\" field. You can use the \"\\n\" character to indicate a newline. Also, do not use any double quotes characters in the \"reason\" field. Your output should be in a valid JSON format.\\nexample output format: {\"choice\": \"<winner of the A/B test>\", \"reason\": \"<your detailed step by step analysis here>\", \"scores\": {\"model_a\": <score for model_a>, \"model_b\": <score for model_b>}}'}, {'role': 'user', 'content': '[Question]\\nWhat are some good references for modernizing a call center?\\n[End of Question]\\n\\n[Response from model_a]\\n1. \"Call Centers For Dummies\" by Real Bergevin, Afshan Kinder, Winston Siegel, and Bruce Simpson: This book provides a comprehensive overview of call center management, including tips for modernization.\\n\\n2. \"The Best Service is No Service: How to Liberate Your Customers from Customer Service, Keep Them Happy, and Control Costs\" by Bill Price and David Jaffe: This book offers innovative strategies for modernizing customer service, including call centers.\\n\\n3. \"Delivering Quality Service: Balancing Customer Perceptions and Expectations\" by Valarie A. Zeithaml, A. Parasuraman, and Leonard L. Berry: This book provides insights into customer expectations and how to meet them in a modern call center.\\n\\n4. \"The Effortless Experience: Conquering the New Battleground for Customer Loyalty\" by Matthew Dixon, Nick Toman, and Rick DeLisi: This book provides strategies for improving customer loyalty, a key aspect of modern call center operations.\\n\\n5. \"Call Center Management on Fast Forward: Succeeding in Today\\'s Dynamic Inbound Environment\" by Brad Cleveland: This book provides a comprehensive guide to managing and modernizing call centers.\\n\\n6. \"The Nordstrom Way to Customer Experience Excellence: Creating a Values-Driven Service Culture\" by Robert Spector and breAnne O. Reeves: This book provides insights into creating a customer-focused culture, a key aspect of modern call centers.\\n\\n7. \"Customer Satisfaction Is Worthless, Customer Loyalty Is Priceless: How to Make Customers Love You, Keep Them Coming Back and Tell Everyone They Know\" by Jeffrey Gitomer: This book provides strategies for building customer loyalty, a key aspect of modern call centers.\\n\\n8. \"Raving Fans: A Revolutionary Approach to Customer Service\" by Ken Blanchard and Sheldon Bowles: This book provides a new approach to customer service, which can be applied to modernizing call centers.\\n\\n9. \"Be Our Guest: Perfecting the Art of Customer Service\" by The Disney Institute: This book provides insights into Disney\\'s approach to customer service, which can be applied to modernizing call centers.\\n\\n10. Websites like Call Centre Helper, Contact Center Pipeline, and Call Centre Management Association provide articles, webinars, and resources on modern call center practices and technologies.\\n\\n11. Research papers and case studies from consulting firms like McKinsey, Deloitte, and Accenture often provide insights into the latest trends and best practices in call center modernization.\\n\\n12. Online courses and certifications from institutions like Coursera, edX, and LinkedIn Learning can provide structured learning experiences on call center management and modernization.\\n[End of Response from model_a]\\n\\n[Response from model_b]\\n There are several resources available that can help you modernize your call center. Here are some references to consider:\\n\\n1. \"The Future of Customer Service: Conversational AI, Automation, and the Role of Human Agents\" by Forrester - This report provides an overview of the current state of customer service and explores the role of conversational AI, automation, and human agents in modern call centers.\\n2. \"The Modern Contact Center: A Guide to Best Practices and Technology\" by ICMI - This guide provides best practices and technology recommendations for modernizing your call center, including strategies for leveraging cloud-based solutions, automation, and analytics.\\n3. \"5 Key Trends Shaping the Future of Customer Service\" by Harvard Business Review - This article highlights five key trends shaping the future of customer service, including the rise of AI, personalization, and the importance of employee engagement.\\n4. \"The Impact of AI on Customer Service: Lessons Learned from Early Adopters\" by MIT Sloan Management Review - This report shares lessons learned from early adopters of AI in customer service, including the benefits and challenges of implementing AI solutions.\\n5. \"The Future of Work in Customer Service: A Journey to the Next-Generation Contact Center\" by Deloitte - This report explores the impact of digital technologies on the customer service industry and provides guidance on how to create a next-generation contact center.\\n6. \"Customer Experience 2020: The Future of Customer Service\" by Oracle - This report examines the future of customer service and provides insights into the technologies and strategies that will shape the industry in the coming years.\\n7. \"The Role of AI in Customer Service: A Global Study\" by NICE - This study explores the use of AI in customer service and provides insights into the benefits and challenges of implementing AI solutions.\\n8. \"The Future of Customer Engagement: Conversational AI, Machine Learning, and the Role of Humans\" by Forbes - This article discusses the role of conversational AI and machine learning in customer engagement and explores the ways in which humans and machines can work together to deliver exceptional customer experiences.\\n9. \"Modernizing the Contact Center: A Guide to Cloud-Based Solutions\" by No Jitter - This guide provides an overview of cloud-based solutions for modernizing the contact center, including the benefits and considerations for migrating to the cloud.\\n10. \"The Future of Customer Service: A Playbook for the Digital Age\" by Salesforce - This playbook provides a comprehensive guide to modernizing your call center, including strategies for leveraging AI, automation, and cloud-based solutions.\\n\\nThese references provide valuable insights and guidance for modernizing your call center and delivering exceptional customer experiences in the digital age.\\n[End of Response from model_b]\\n\\nPlease complete the A/B test. Make sure that your entire response is a valid JSON string.'}], 'user': '76314a10-f32a-4188-bba8-fd56f0ba95f4_56a5bedc-823e-4193-af89-d1caa000f806', 'temperature': 0.2, 'max_tokens': 1024}, {'id': 'chatcmpl-7gjq8012Wyjpc03B44haC8MxbXlDV', 'object': 'chat.completion', 'created': 1690420340, 'model': 'gpt-4-0613', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': '{\"choice\": \"model_a\", \"reason\": \"In step 1, I read the question which was about finding good references for modernizing a call center. In step 2, I read the responses from both models. Both models provided a list of resources, including books, reports, and guides. However, model_a\\'s response was more comprehensive and diverse, including books, websites, research papers, and online courses. This makes it more useful for someone looking for a wide range of resources. In step 3, I analyzed both responses based on accuracy, safety, completeness, usefulness, and readability. Both responses were accurate, safe, and readable. However, model_a\\'s response was more complete and useful, as it provided a wider range of resources. Therefore, I assigned a score of 9 to model_a and a score of 8 to model_b. In step 4, I compared the scores and chose model_a as the winner because it had the highest score.\", \"scores\": {\"model_a\": 9, \"model_b\": 8}}'}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 1716, 'completion_tokens': 216, 'total_tokens': 1932}}]\n"
     ]
    }
   ],
   "source": [
    "results, failed_lines = read_jsonl('requests_results.jsonl')\n",
    "\n",
    "if len(failed_lines):\n",
    "    print(f\"{len(failed_lines)} failed lines read from file\")\n",
    "    print(f\"Example: {failed_lines[0]}\")\n",
    "\n",
    "print(f\"{len(results)} results read from file\")\n",
    "if len(results) > 0:\n",
    "    print(f\"Example: {results[0]}\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:37.509153Z",
     "start_time": "2023-07-27T01:12:37.497220Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 eval entries created\n",
      "Example: {'ab_test_id': UUID('76314a10-f32a-4188-bba8-fd56f0ba95f4'), 'model_a': UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'model_b': UUID('db8d724d-710b-4926-8e7c-b4baa9b71901'), 'prompt_id': UUID('56a5bedc-823e-4193-af89-d1caa000f806'), 'submitted_by': UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'selected_model': UUID('10596d4c-4bbe-4287-b781-c4f04a13ab5d'), 'additional_feedback': \"In step 1, I read the question which was about finding good references for modernizing a call center. In step 2, I read the responses from both models. Both models provided a list of resources, including books, reports, and guides. However, model_a's response was more comprehensive and diverse, including books, websites, research papers, and online courses. This makes it more useful for someone looking for a wide range of resources. In step 3, I analyzed both responses based on accuracy, safety, completeness, usefulness, and readability. Both responses were accurate, safe, and readable. However, model_a's response was more complete and useful, as it provided a wider range of resources. Therefore, I assigned a score of 9 to model_a and a score of 8 to model_b. In step 4, I compared the scores and chose model_a as the winner because it had the highest score.\", 'model_a_score': 9, 'model_b_score': 8, 'submitted_at': datetime.datetime(2023, 7, 26, 18, 12, 20)}\n"
     ]
    }
   ],
   "source": [
    "games_lookup = games_list_to_lookup(games)\n",
    "\n",
    "eval_entries = [parse_result_to_eval_entry(result, games_lookup) for result in results]\n",
    "print(f\"{len(eval_entries)} eval entries created\")\n",
    "if len(eval_entries) > 0:\n",
    "    print(f\"Example: {eval_entries[0]}\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:37.514676Z",
     "start_time": "2023-07-27T01:12:37.511089Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 eval entries inserted into db\n"
     ]
    }
   ],
   "source": [
    "await insert_evals_by_model_into_db(eval_entries)\n",
    "print(f\"{len(eval_entries)} eval entries inserted into db\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2023-07-27T01:12:37.582542Z",
     "start_time": "2023-07-27T01:12:37.517603Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}