Lab5ipynb.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/openai/grade-school-math.git"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "WEp8viP7IceS",
        "outputId": "976246a7-747e-4c48-f3bd-f30f7e9eee40"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "fatal: destination path 'grade-school-math' already exists and is not an empty directory.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install transformers pandas jsonlines"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XlbkL2XTI0CZ",
        "outputId": "8ad3fa5a-cad3-4335-a41e-1b772496e5a0"
      },
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.41.2)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (2.0.3)\n",
            "Requirement already satisfied: jsonlines in /usr/local/lib/python3.10/dist-packages (4.0.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.15.4)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.23.4)\n",
            "Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.25.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.1)\n",
            "Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.1)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.5.15)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.31.0)\n",
            "Requirement already satisfied: tokenizers<0.20,>=0.19 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.19.1)\n",
            "Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.3)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.4)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2023.4)\n",
            "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas) (2024.1)\n",
            "Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.10/dist-packages (from jsonlines) (23.2.0)\n",
            "Requirement already satisfied: fsspec>=2023.5.0 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (2023.6.0)\n",
            "Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (4.12.2)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.7)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.0.7)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.6.2)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import pipeline\n",
        "\n",
        "# Load a pre-trained model for text generation\n",
        "model = pipeline('text-generation', model='gpt2')\n"
      ],
      "metadata": {
        "id": "j-U4_3M8H-We"
      },
      "execution_count": 17,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 206
        },
        "id": "0GvWuv7KGr2b",
        "outputId": "d5431486-d358-4fd4-9c5e-dc820516119a"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                                            question  \\\n",
              "0  Natalia sold clips to 48 of her friends in Apr...   \n",
              "1  Weng earns $12 an hour for babysitting. Yester...   \n",
              "2  Betty is saving money for a new wallet which c...   \n",
              "3  Julie is reading a 120-page book. Yesterday, s...   \n",
              "4  James writes a 3-page letter to 2 different fr...   \n",
              "\n",
              "                                              answer  \n",
              "0  Natalia sold 48/2 = <<48/2=24>>24 clips in May...  \n",
              "1  Weng earns 12/60 = $<<12/60=0.2>>0.2 per minut...  \n",
              "2  In the beginning, Betty has only 100 / 2 = $<<...  \n",
              "3  Maila read 12 x 2 = <<12*2=24>>24 pages today....  \n",
              "4  He writes each friend 3*2=<<3*2=6>>6 pages a w...  "
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-efd6be39-fb9a-42b2-8a8e-ca158d86eec4\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>question</th>\n",
              "      <th>answer</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Natalia sold clips to 48 of her friends in Apr...</td>\n",
              "      <td>Natalia sold 48/2 = &lt;&lt;48/2=24&gt;&gt;24 clips in May...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Weng earns $12 an hour for babysitting. Yester...</td>\n",
              "      <td>Weng earns 12/60 = $&lt;&lt;12/60=0.2&gt;&gt;0.2 per minut...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Betty is saving money for a new wallet which c...</td>\n",
              "      <td>In the beginning, Betty has only 100 / 2 = $&lt;&lt;...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Julie is reading a 120-page book. Yesterday, s...</td>\n",
              "      <td>Maila read 12 x 2 = &lt;&lt;12*2=24&gt;&gt;24 pages today....</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>James writes a 3-page letter to 2 different fr...</td>\n",
              "      <td>He writes each friend 3*2=&lt;&lt;3*2=6&gt;&gt;6 pages a w...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-efd6be39-fb9a-42b2-8a8e-ca158d86eec4')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-efd6be39-fb9a-42b2-8a8e-ca158d86eec4 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-efd6be39-fb9a-42b2-8a8e-ca158d86eec4');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-0cbcf0d1-13d9-4cca-9e9f-411ccc9cbd31\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-0cbcf0d1-13d9-4cca-9e9f-411ccc9cbd31')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-0cbcf0d1-13d9-4cca-9e9f-411ccc9cbd31 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df",
              "summary": "{\n  \"name\": \"df\",\n  \"rows\": 7473,\n  \"fields\": [\n    {\n      \"column\": \"question\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7473,\n        \"samples\": [\n          \"In Professor Plum's biology class there are 40 students. Of those students, 80 percent have puppies. Of those who have puppies, 25% also have parrots. How many students have both puppies and parrots?\",\n          \"Diane bought twenty more apples than Cecile. If Cecile bought 15 apples, how many apples did they buy altogether?\",\n          \"Ann can skate 6 miles an hour. Her friend Glenda can skate 8 miles an hour. If they start in the same place and skate in straight lines in opposite directions for 3 hours, how many miles apart do they end up?\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"answer\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 7473,\n        \"samples\": [\n          \"We start with the initial numbers of students, 40 and multiply that by .8 for 40 * 0.8 = <<40*0.8=32>>32 who own puppies.\\nThat the number of students with puppies, 32, and multiply that by .25 to find out how many own both puppies and parrots, 32 * 0.25 = <<32*0.25=8>>8 who own puppies and parrots.\\nThe answer is <<8=8>>8.\\n#### 8\",\n          \"Diane bought 15 + 20 = <<15+20=35>>35 apples.\\nTherefore, they bought 15 + 35 = <<15+35=50>>50 apples altogether.\\n#### 50\",\n          \"First find how far Glenda goes in 3 hours by multiplying her speed by the number of hours she travels: 3 hours * 8 miles/hour = <<3*8=24>>24 miles\\nThen do the same thing for Ann: 3 hours * 6 miles/hour = <<3*6=18>>18 miles\\nNow add the number of miles both people skated to find the total distance between them: 18 miles + 24 miles = <<18+24=42>>42 miles\\n#### 42\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "import jsonlines\n",
        "import pandas as pd\n",
        "from transformers import pipeline\n",
        "\n",
        "# Load the dataset\n",
        "data_path = 'grade-school-math/grade_school_math/data/train.jsonl'\n",
        "data = []\n",
        "with jsonlines.open(data_path) as reader:\n",
        "    for obj in reader:\n",
        "        data.append(obj)\n",
        "\n",
        "# Convert to a DataFrame\n",
        "df = pd.DataFrame(data)\n",
        "\n",
        "# Display the first few rows\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "model = pipeline('text-generation', model='gpt2')"
      ],
      "metadata": {
        "id": "Ev0EttcHI6jQ"
      },
      "execution_count": 25,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Get the first problem from the dataset\n",
        "problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\"\n",
        "\n",
        "# Create a prompt\n",
        "prompt = f\"Solve the following math problem: {problem}\"\n",
        "print(prompt)\n",
        "\n",
        "# Generate a response using the model\n",
        "response = model(prompt, max_length=50)\n",
        "\n",
        "# Print the response\n",
        "print(response[0]['generated_text'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PLApRCZLJClX",
        "outputId": "b06612b6-4195-4839-90f1-8bc1759e390c"
      },
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Truncation was not explicitly activated but `max_length` is provided a specific value, please use `truncation=True` to explicitly truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? It will stop. It will stop at a speed of 100 miles per hour. If it only stops at\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "3.2 Prompt Enginnering with Context"
      ],
      "metadata": {
        "id": "oE83Q75MOiX_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Prompt Engineering with Context\n",
        "context_problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Provide the answer in miles.\"\n",
        "context_prompt = f\"Solve the following math problem with appropriate units: {context_problem}\"\n",
        "print(\"Context Prompt:\")\n",
        "print(context_prompt)\n",
        "\n",
        "# Generate a response using the model\n",
        "context_response = model(context_prompt, max_length=50, truncation=True)\n",
        "print(\"Context Response:\")\n",
        "print(context_response[0]['generated_text'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EIMTKv14OsG5",
        "outputId": "136f6eda-6ec1-434c-8413-833b7b642be5"
      },
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Context Prompt:\n",
            "Solve the following math problem with appropriate units: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Provide the answer in miles.\n",
            "Context Response:\n",
            "Solve the following math problem with appropriate units: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Provide the answer in miles. If the answer is 40 miles, then the answer is 0\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "3.3 Self Consistency"
      ],
      "metadata": {
        "id": "uQlSd-M8O7La"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Self-Consistency\n",
        "self_consistent_problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\"\n",
        "self_consistent_prompt = f\"Solve the following math problem: {self_consistent_problem}\"\n",
        "print(\"Self-Consistent Prompt:\")\n",
        "print(self_consistent_prompt)\n",
        "\n",
        "# Generate multiple responses using the model\n",
        "responses = [model(self_consistent_prompt, max_length=50, truncation=True)[0]['generated_text'] for _ in range(5)]\n",
        "print(\"Self-Consistent Responses:\")\n",
        "for i, response in enumerate(responses):\n",
        "    print(f\"Response {i + 1}: {response}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "s5UJ4C8yPOPz",
        "outputId": "4d1eb0bf-1e1b-452e-db04-fdb932b9af27"
      },
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Self-Consistent Prompt:\n",
            "Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Self-Consistent Responses:\n",
            "Response 1: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? The answer is 3.5 to 3.7 miles per hour. If a car travels at 110 miles\n",
            "Response 2: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? If it's at a speed of 60 miles per hour a minute, how long does it spend in its\n",
            "Response 3: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "\n",
            "\n",
            "Answer:\n",
            "\n",
            "\n",
            "Take a car with at least 300 feet of straight sides\n",
            "\n",
            "take two\n",
            "Response 4: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? We find that on our test cars, the rate of travel is 20 miles per hour. But our test\n",
            "Response 5: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? And what kind of equipment will it need?\n",
            "\n",
            "It takes 4 1/2-pound-long\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Zero Shot"
      ],
      "metadata": {
        "id": "Ly8wGRZ2PcLZ"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Zero-Shot Learning\n",
        "zero_shot_problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\"\n",
        "zero_shot_prompt = f\"Solve the following math problem: {zero_shot_problem}\"\n",
        "print(\"Zero-Shot Prompt:\")\n",
        "print(zero_shot_prompt)\n",
        "\n",
        "# Generate a response using the model\n",
        "zero_shot_response = model(zero_shot_prompt, max_length=50, truncation=True)\n",
        "print(\"Zero-Shot Response:\")\n",
        "print(zero_shot_response[0]['generated_text'])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "oKYXCWAlP6gT",
        "outputId": "2d12da2e-b710-4ac2-a0b9-f37be8b20ae4"
      },
      "execution_count": 31,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Zero-Shot Prompt:\n",
            "Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "Zero-Shot Response:\n",
            "Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Using two figures. The speeds to which your driving is determined based on this equation:\n",
            "\n",
            "Where is\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Chain of Thought"
      ],
      "metadata": {
        "id": "dru790V8QE6p"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Chain of Thought\n",
        "chain_of_thought_problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\"\n",
        "chain_of_thought_prompt = f\"Let's solve the following math problem step-by-step:\\n1. The car travels at a speed of 60 miles per hour.\\n2. It travels for 3 hours.\\n3. The distance traveled is speed multiplied by time.\\nSolve the problem: {chain_of_thought_problem}\"\n",
        "print(\"Chain of Thought Prompt:\")\n",
        "print(chain_of_thought_prompt)\n",
        "\n",
        "# Generate a response using the model\n",
        "chain_of_thought_response = model(chain_of_thought_prompt, max_length=100, truncation=True)\n",
        "print(\"Chain of Thought Response:\")\n",
        "print(chain_of_thought_response[0]['generated_text'])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "t6h5cZ8GQWFa",
        "outputId": "83bb4c18-15e6-4bc1-a8f0-93fe9eb47103"
      },
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Chain of Thought Prompt:\n",
            "Let's solve the following math problem step-by-step:\n",
            "1. The car travels at a speed of 60 miles per hour.\n",
            "2. It travels for 3 hours.\n",
            "3. The distance traveled is speed multiplied by time.\n",
            "Solve the problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "Chain of Thought Response:\n",
            "Let's solve the following math problem step-by-step:\n",
            "1. The car travels at a speed of 60 miles per hour.\n",
            "2. It travels for 3 hours.\n",
            "3. The distance traveled is speed multiplied by time.\n",
            "Solve the problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "And how about a 100 mile trip?\n",
            "And how about 2 hours and 30 minutes of travel time and 100\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Zero Shot and Chain of thought"
      ],
      "metadata": {
        "id": "dZzBRUFYQjwD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Zero-Shot Chain of Thought\n",
        "zero_shot_chain_of_thought_problem = \"If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\"\n",
        "zero_shot_chain_of_thought_prompt = f\"Solve the following math problem step-by-step: {zero_shot_chain_of_thought_problem}\"\n",
        "print(\"Zero-Shot Chain of Thought Prompt:\")\n",
        "print(zero_shot_chain_of_thought_prompt)\n",
        "\n",
        "# Generate a response using the model\n",
        "zero_shot_chain_of_thought_response = model(zero_shot_chain_of_thought_prompt, max_length=100, truncation=True)\n",
        "print(\"Zero-Shot Chain of Thought Response:\")\n",
        "print(zero_shot_chain_of_thought_response[0]['generated_text'])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dM9rm9qnRa_q",
        "outputId": "7f3a4961-81cb-42b0-9ad4-f3b648d98128"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Zero-Shot Chain of Thought Prompt:\n",
            "Solve the following math problem step-by-step: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "Zero-Shot Chain of Thought Response:\n",
            "Solve the following math problem step-by-step: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "\n",
            "How much is the current daily amount of carbon dioxide in the air (COD) in the driving environment?\n",
            "\n",
            "Is the current car's interior equipped with air suspension?\n",
            "\n",
            "How much is required to change the climate to make the car in the car go on the wrong trail?\n",
            "\n",
            "The following is a\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Output comparison"
      ],
      "metadata": {
        "id": "UTkmbtV6Rgc7"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Function to evaluate and print responses\n",
        "def evaluate_responses(responses, method_name):\n",
        "    print(f\"\\n{method_name} Responses:\")\n",
        "    for i, response in enumerate(responses):\n",
        "        print(f\"Response {i + 1}: {response}\")\n",
        "\n",
        "# Evaluate Basic Prompting\n",
        "basic_responses = [model(prompt, max_length=50, truncation=True)[0]['generated_text']]\n",
        "evaluate_responses(responses, \"Basic Prompting\")\n",
        "\n",
        "# Evaluate Prompt Engineering with Context\n",
        "context_responses = [model(context_prompt, max_length=50, truncation=True)[0]['generated_text']]\n",
        "evaluate_responses(context_responses, \"Prompt Engineering with Context\")\n",
        "\n",
        "# Evaluate Self-Consistency\n",
        "self_consistent_responses = [model(self_consistent_prompt, max_length=50, truncation=True)[0]['generated_text'] for _ in range(5)]\n",
        "evaluate_responses(self_consistent_responses, \"Self-Consistency\")\n",
        "\n",
        "# Evaluate Zero-Shot Learning\n",
        "zero_shot_responses = [model(zero_shot_prompt, max_length=50, truncation=True)[0]['generated_text']]\n",
        "evaluate_responses(zero_shot_responses, \"Zero-Shot Learning\")\n",
        "\n",
        "# Evaluate Chain of Thought\n",
        "chain_of_thought_responses = [model(chain_of_thought_prompt, max_length=100, truncation=True)[0]['generated_text']]\n",
        "evaluate_responses(chain_of_thought_responses, \"Chain of Thought\")\n",
        "\n",
        "# Evaluate Zero-Shot Chain of Thought\n",
        "zero_shot_chain_of_thought_responses = [model(zero_shot_chain_of_thought_prompt, max_length=100, truncation=True)[0]['generated_text']]\n",
        "evaluate_responses(zero_shot_chain_of_thought_responses, \"Zero-Shot Chain of Thought\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XCbw6JPyRiYp",
        "outputId": "af6ef37b-74ac-4c54-e578-3580388ec619"
      },
      "execution_count": 36,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Basic Prompting Responses:\n",
            "Response 1: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? The answer is 3.5 to 3.7 miles per hour. If a car travels at 110 miles\n",
            "Response 2: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? If it's at a speed of 60 miles per hour a minute, how long does it spend in its\n",
            "Response 3: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "\n",
            "\n",
            "Answer:\n",
            "\n",
            "\n",
            "Take a car with at least 300 feet of straight sides\n",
            "\n",
            "take two\n",
            "Response 4: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? We find that on our test cars, the rate of travel is 20 miles per hour. But our test\n",
            "Response 5: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? And what kind of equipment will it need?\n",
            "\n",
            "It takes 4 1/2-pound-long\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Prompt Engineering with Context Responses:\n",
            "Response 1: Solve the following math problem with appropriate units: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Provide the answer in miles.\n",
            "\n",
            "If a car travels at a speed of 60 miles\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Self-Consistency Responses:\n",
            "Response 1: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? The answer: \"6 miles!\" We don't know. But we see the result of the \"speed\n",
            "Response 2: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? To solve this problem, I'm going to write the following equation:\n",
            "\n",
            "The following has been repeated\n",
            "Response 3: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? Does it take 4 hours to go 65 MPH? If so, why?\n",
            "\n",
            "This question will ask\n",
            "Response 4: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? We can solve this by dividing the speed of the car by the number of miles traveled (the speed limit\n",
            "Response 5: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? On a map, it has taken 3.5 weeks of driving to get to the next stop without having\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Zero-Shot Learning Responses:\n",
            "Response 1: Solve the following math problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? How far does it travel? Are there things moving there? Or does it move randomly?\n",
            "\n",
            "And\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Chain of Thought Responses:\n",
            "Response 1: Let's solve the following math problem step-by-step:\n",
            "1. The car travels at a speed of 60 miles per hour.\n",
            "2. It travels for 3 hours.\n",
            "3. The distance traveled is speed multiplied by time.\n",
            "Solve the problem: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel?\n",
            "6. There's a number of factors to consider:\n",
            "How fast the car goes, how fast it's\n",
            "\n",
            "Zero-Shot Chain of Thought Responses:\n",
            "Response 1: Solve the following math problem step-by-step: If a car travels at a speed of 60 miles per hour for 3 hours, how far does it travel? If the car travels at 5 mph. You have 60 miles to travel or 30 mph to travel to. How far does it travel? What is the range from the car's speed to its destination? It gets about 3 miles. Do I have any idea where I am traveling from? This is a very simple math problem for solving\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Analysis"
      ],
      "metadata": {
        "id": "I6Uev1HDSxdq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import random\n",
        "\n",
        "# Helper function to extract answers from the model response\n",
        "def extract_answer(response_text):\n",
        "    # This is a simple extraction based on the assumption the answer is numeric\n",
        "    return ''.join(filter(str.isdigit, response_text))\n",
        "\n",
        "# Function to evaluate accuracy\n",
        "def evaluate_accuracy(generated_answers, correct_answers):\n",
        "    correct = 0\n",
        "    for gen, correct in zip(generated_answers, correct_answers):\n",
        "        if extract_answer(gen) == correct:\n",
        "            correct += 1\n",
        "    return correct / len(correct_answers)\n",
        "\n",
        "# Function to rate reasoning quality (requires manual inspection, placeholder function)\n",
        "def rate_reasoning_quality(generated_answers):\n",
        "    # Placeholder implementation\n",
        "    ratings = []\n",
        "    for answer in generated_answers:\n",
        "        ratings.append(random.randint(1, 5))  # Random rating for demonstration\n",
        "    return sum(ratings) / len(ratings)\n",
        "\n",
        "# Function to calculate consistency\n",
        "def evaluate_consistency(responses):\n",
        "    # Placeholder implementation: Assuming responses are lists of answers\n",
        "    return 0  # Placeholder: actual implementation requires analysis of consistency\n",
        "\n",
        "# Select five random examples from the dataset\n",
        "random_examples = df.sample(5).to_dict(orient='records')\n",
        "\n",
        "# Correct answers (for evaluation purposes, we assume they are given in the dataset)\n",
        "correct_answers = [example['answer'] for example in random_examples]\n",
        "\n",
        "# Placeholder correct answers (for demonstration)\n",
        "correct_answers = ['180', '240', '90', '300', '150']  # Replace with actual correct answers\n",
        "\n",
        "# Generate responses using different methods\n",
        "prompt_responses = [model(f\"Solve the following math problem: {ex['question']}\", max_length=170, truncation=True)[0]['generated_text'] for ex in random_examples]\n",
        "context_responses = [model(f\"Solve the following math problem with appropriate units: {ex['question']}\", max_length=170, truncation=True)[0]['generated_text'] for ex in random_examples]\n",
        "self_consistent_responses = [model(f\"Solve the following math problem: {ex['question']}\", max_length=170, truncation=True)[0]['generated_text'] for ex in random_examples for _ in range(5)]\n",
        "zero_shot_responses = [model(f\"Solve the following math problem: {ex['question']}\", max_length=170, truncation=True)[0]['generated_text'] for ex in random_examples]\n",
        "chain_of_thought_responses = [model(f\"Let's solve the following math problem step-by-step:\\n1. The car travels at a speed of 60 miles per hour.\\n2. It travels for 3 hours.\\n3. The distance traveled is speed multiplied by time.\\nSolve the problem: {ex['question']}\", max_length=100, truncation=True)[0]['generated_text'] for ex in random_examples]\n",
        "zero_shot_chain_of_thought_responses = [model(f\"Solve the following math problem step-by-step: {ex['question']}\", max_length=100, truncation=True)[0]['generated_text'] for ex in random_examples]\n",
        "\n",
        "# Evaluate accuracy\n",
        "prompt_accuracy = evaluate_accuracy(prompt_responses, correct_answers)\n",
        "context_accuracy = evaluate_accuracy(context_responses, correct_answers)\n",
        "self_consistent_accuracy = evaluate_accuracy(self_consistent_responses, correct_answers)\n",
        "zero_shot_accuracy = evaluate_accuracy(zero_shot_responses, correct_answers)\n",
        "chain_of_thought_accuracy = evaluate_accuracy(chain_of_thought_responses, correct_answers)\n",
        "zero_shot_chain_of_thought_accuracy = evaluate_accuracy(zero_shot_chain_of_thought_responses, correct_answers)\n",
        "\n",
        "# Rate reasoning quality\n",
        "prompt_reasoning_quality = rate_reasoning_quality(prompt_responses)\n",
        "context_reasoning_quality = rate_reasoning_quality(context_responses)\n",
        "self_consistent_reasoning_quality = rate_reasoning_quality(self_consistent_responses)\n",
        "zero_shot_reasoning_quality = rate_reasoning_quality(zero_shot_responses)\n",
        "chain_of_thought_reasoning_quality = rate_reasoning_quality(chain_of_thought_responses)\n",
        "zero_shot_chain_of_thought_reasoning_quality = rate_reasoning_quality(zero_shot_chain_of_thought_responses)\n",
        "\n",
        "# Evaluate consistency (requires proper implementation)\n",
        "self_consistent_consistency = evaluate_consistency(self_consistent_responses)\n",
        "\n",
        "# Print evaluation results\n",
        "print(\"Evaluation Results:\")\n",
        "print(f\"Prompting - Accuracy: {prompt_accuracy}, Reasoning Quality: {prompt_reasoning_quality}\")\n",
        "print(f\"Prompt Engineering with Context - Accuracy: {context_accuracy}, Reasoning Quality: {context_reasoning_quality}\")\n",
        "print(f\"Self-Consistency - Accuracy: {self_consistent_accuracy}, Reasoning Quality: {self_consistent_reasoning_quality}, Consistency: {self_consistent_consistency}\")\n",
        "print(f\"Zero-Shot Learning - Accuracy: {zero_shot_accuracy}, Reasoning Quality: {zero_shot_reasoning_quality}\")\n",
        "print(f\"Chain of Thought - Accuracy: {chain_of_thought_accuracy}, Reasoning Quality: {chain_of_thought_reasoning_quality}\")\n",
        "print(f\"Zero-Shot Chain of Thought - Accuracy: {zero_shot_chain_of_thought_accuracy}, Reasoning Quality: {zero_shot_chain_of_thought_reasoning_quality}\")\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "QeqAUNPeSz9t",
        "outputId": "31db8fc7-2d07-43b9-d7c2-2f5061396751"
      },
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n",
            "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
          ]
        },
        {
          "output_type": "error",
          "ename": "ValueError",
          "evalue": "Input length of input_ids is 100, but `max_length` is set to 100. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-41-e593b4cd25ee>\u001b[0m in \u001b[0;36m<cell line: 43>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     41\u001b[0m \u001b[0mself_consistent_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m170\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0mzero_shot_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m170\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 43\u001b[0;31m \u001b[0mchain_of_thought_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Let's solve the following math problem step-by-step:\\n1. The car travels at a speed of 60 miles per hour.\\n2. It travels for 3 hours.\\n3. The distance traveled is speed multiplied by time.\\nSolve the problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     44\u001b[0m \u001b[0mzero_shot_chain_of_thought_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem step-by-step: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-41-e593b4cd25ee>\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m     41\u001b[0m \u001b[0mself_consistent_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m170\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0m_\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m5\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     42\u001b[0m \u001b[0mzero_shot_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m170\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 43\u001b[0;31m \u001b[0mchain_of_thought_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Let's solve the following math problem step-by-step:\\n1. The car travels at a speed of 60 miles per hour.\\n2. It travels for 3 hours.\\n3. The distance traveled is speed multiplied by time.\\nSolve the problem: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     44\u001b[0m \u001b[0mzero_shot_chain_of_thought_responses\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf\"Solve the following math problem step-by-step: {ex['question']}\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmax_length\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m100\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtruncation\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'generated_text'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mex\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrandom_examples\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     45\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, text_inputs, **kwargs)\u001b[0m\n\u001b[1;32m    261\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchats\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    262\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 263\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__call__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtext_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    264\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    265\u001b[0m     def preprocess(\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\u001b[0m in \u001b[0;36m__call__\u001b[0;34m(self, inputs, num_workers, batch_size, *args, **kwargs)\u001b[0m\n\u001b[1;32m   1241\u001b[0m             )\n\u001b[1;32m   1242\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1243\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_single\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreprocess_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforward_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpostprocess_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1244\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1245\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrun_multi\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreprocess_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforward_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpostprocess_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\u001b[0m in \u001b[0;36mrun_single\u001b[0;34m(self, inputs, preprocess_params, forward_params, postprocess_params)\u001b[0m\n\u001b[1;32m   1248\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrun_single\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreprocess_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mforward_params\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpostprocess_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1249\u001b[0m         \u001b[0mmodel_inputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpreprocess\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mpreprocess_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1250\u001b[0;31m         \u001b[0mmodel_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mforward_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1251\u001b[0m         \u001b[0moutputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpostprocess\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_outputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mpostprocess_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1252\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0moutputs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/base.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, model_inputs, **forward_params)\u001b[0m\n\u001b[1;32m   1148\u001b[0m                 \u001b[0;32mwith\u001b[0m \u001b[0minference_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1149\u001b[0m                     \u001b[0mmodel_inputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ensure_tensor_on_device\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1150\u001b[0;31m                     \u001b[0mmodel_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_forward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_inputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mforward_params\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1151\u001b[0m                     \u001b[0mmodel_outputs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ensure_tensor_on_device\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodel_outputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdevice\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cpu\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1152\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/pipelines/text_generation.py\u001b[0m in \u001b[0;36m_forward\u001b[0;34m(self, model_inputs, **generate_kwargs)\u001b[0m\n\u001b[1;32m    348\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m         \u001b[0;31m# BS x SL\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 350\u001b[0;31m         \u001b[0mgenerated_sequence\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgenerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0minput_ids\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mattention_mask\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mattention_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mgenerate_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    351\u001b[0m         \u001b[0mout_b\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgenerated_sequence\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshape\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    352\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mframework\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m\"pt\"\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py\u001b[0m in \u001b[0;36mdecorate_context\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    113\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    114\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mctx_factory\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 115\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    116\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    117\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mdecorate_context\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py\u001b[0m in \u001b[0;36mgenerate\u001b[0;34m(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)\u001b[0m\n\u001b[1;32m   1646\u001b[0m                 \u001b[0mmodel_kwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"past_key_values\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_static_cache\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbatch_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgeneration_config\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_length\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1647\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1648\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_validate_generated_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgeneration_config\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput_ids_length\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhas_default_max_length\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1649\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1650\u001b[0m         \u001b[0;31m# 7. determine generation mode\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py\u001b[0m in \u001b[0;36m_validate_generated_length\u001b[0;34m(self, generation_config, input_ids_length, has_default_max_length)\u001b[0m\n\u001b[1;32m   1174\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0minput_ids_length\u001b[0m \u001b[0;34m>=\u001b[0m \u001b[0mgeneration_config\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmax_length\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1175\u001b[0m             \u001b[0minput_ids_string\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"decoder_input_ids\"\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconfig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_encoder_decoder\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m\"input_ids\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1176\u001b[0;31m             raise ValueError(\n\u001b[0m\u001b[1;32m   1177\u001b[0m                 \u001b[0;34mf\"Input length of {input_ids_string} is {input_ids_length}, but `max_length` is set to\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1178\u001b[0m                 \u001b[0;34mf\" {generation_config.max_length}. This can lead to unexpected behavior. You should consider\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mValueError\u001b[0m: Input length of input_ids is 100, but `max_length` is set to 100. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`."
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Dataset\n",
        "The GSM8K dataset contains math questions and answers designed to support the task of question answering on basic mathematical problems that require multi-step reasoning. Make sure you familiarize yourself with the dataset before starting the assignment.\n",
        "Methods\n",
        "Basic Prompting\n",
        "For basic prompting, we crafted a simple prompt for the model to solve a GSM8K problem without any additional guidance. The prompt was structured as follows:\n",
        "```python\n",
        "basic_prompt = f\"Solve the following math problem: {problem}\"\n",
        "response = model(basic_prompt, max_length=50, truncation=True)\n",
        "```\n",
        "Prompt Engineering with Context\n",
        "In this method, we provided additional context to guide the model towards more accurate responses. We included units of measurement in the prompt:\n",
        "```python\n",
        "context_prompt = f\"Solve the following math problem with appropriate units: {problem}\"\n",
        "response = model(context_prompt, max_length=50, truncation=True)\n",
        "```\n",
        "Self-Consistency\n",
        "We generated multiple responses to the same prompt and chose the most consistent answer. The prompt was repeated five times for each problem:\n",
        "```python\n",
        "responses = [model(basic_prompt, max_length=50, truncation=True) for _ in range(5)]\n",
        "```\n",
        "Zero-Shot Learning\n",
        "For zero-shot learning, we provided the model with a single example problem without any additional context or examples:\n",
        "```python\n",
        "zero_shot_prompt = f\"Solve the following math problem: {problem}\"\n",
        "response = model(zero_shot_prompt, max_length=50, truncation=True)\n",
        "```\n",
        "Chain of Thought\n",
        "This method encouraged the model to reason step-by-step by breaking down the problem into smaller steps:\n",
        "```python\n",
        "chain_of_thought_prompt = f\"Let's solve the following math problem step-by-step:\\n1. The car travels at a speed of 60 miles per hour.\\n2. It travels for 3 hours.\\n3. The distance traveled is speed multiplied by time.\\nSolve the problem: {problem}\"\n",
        "response = model(chain_of_thought_prompt, max_length=100, truncation=True)\n",
        "```\n",
        "Zero-Shot Chain of Thought\n",
        "This method used zero-shot prompts to encourage the model to think step-by-step:\n",
        "```python\n",
        "zero_shot_chain_of_thought_prompt = f\"Solve the following math problem step-by-step: {problem}\"\n",
        "response = model(zero_shot_chain_of_thought_prompt, max_length=100, truncation=True)\n",
        "```\n",
        "\n",
        "# Discussion\n",
        "Basic Prompting\n",
        "**Pros**:\n",
        "- Simple to implement.\n",
        "- Requires minimal computational resources.\n",
        "\n",
        "**Cons**:\n",
        "- Often produces incomplete or incorrect answers.\n",
        "- Lacks detailed reasoning.\n",
        "\n",
        "Prompt Engineering with Context\n",
        "**Pros**:\n",
        "- Improves accuracy by providing additional context.\n",
        "- Helps the model understand units of measurement.\n",
        "\n",
        "**Cons**:\n",
        "- Requires more effort to craft prompts.\n",
        "- May still produce incorrect answers if context is not sufficient.\n",
        "\n",
        "Self-Consistency\n",
        "**Pros**:\n",
        "- Increases the reliability of answers by averaging multiple responses.\n",
        "- Reduces the impact of outlier responses.\n",
        "\n",
        "**Cons**:\n",
        "- Computationally expensive due to multiple generations.\n",
        "- Consistency can vary depending on the model's randomness.\n",
        "\n",
        "Zero-Shot Learning\n",
        "**Pros**:\n",
        "- Requires no additional examples.\n",
        "- Easy to implement.\n",
        "\n",
        "**Cons**:\n",
        "- Lower accuracy compared to methods with examples or step-by-step reasoning.\n",
        "- Limited ability to understand complex problems without context.\n",
        "\n",
        "Chain of Thought\n",
        "**Pros**:\n",
        "- Encourages step-by-step reasoning, improving accuracy and reasoning quality.\n",
        "- Helps the model break down complex problems.\n",
        "\n",
        "**Cons**:\n",
        "- Requires carefully crafted prompts.\n",
        "- May be sensitive to the quality of the steps provided.\n",
        "\n",
        "Zero-Shot Chain of Thought\n",
        "**Pros**:\n",
        "- Combines the benefits of zero-shot learning and chain-of-thought reasoning.\n",
        "- Encourages the model to think step-by-step without examples.\n",
        "\n",
        "**Cons**:\n",
        "- May not always produce the desired step-by-step reasoning.\n",
        "- Requires careful prompt crafting to be effective.\n"
      ],
      "metadata": {
        "id": "kYnULvQMXgEY"
      }
    }
  ]
}