ai-engineering/example2.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "When training or fine-tuning an LLM, model builders utilize certain **Prompt Formats**. These look like a special markup type of language that allows the prompt curators to specify things like chat conversation turns, end-of-statements, tool calls, etc.\n",
        "\n",
        "If you download an open model and try to prompt the without utilizing the expected prompt format, you will possibly get poor performance. Here we will demonstrate that along with application of the prompt format. However, AI platforms usually provide a way to automatically apply the appropriate prompt format via a \"chat\" endpoint. This doesn't mean you have to use this only for chat, but it is a convenient way to manage the proper prompt formatting."
      ],
      "metadata": {
        "id": "o14TvGScYAqZ"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Dependencies and imports"
      ],
      "metadata": {
        "id": "Jc-nVEbsX8bU"
      }
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "u9pot_Yc2FMw",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "13ef63e0-560c-4c05-b28b-2e46d10a608a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting predictionguard\n",
            "  Downloading predictionguard-2.7.0-py2.py3-none-any.whl.metadata (872 bytes)\n",
            "Requirement already satisfied: tabulate>=0.8.10 in /usr/local/lib/python3.10/dist-packages (from predictionguard) (0.9.0)\n",
            "Requirement already satisfied: requests>=2.27.1 in /usr/local/lib/python3.10/dist-packages (from predictionguard) (2.32.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests>=2.27.1->predictionguard) (3.4.0)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests>=2.27.1->predictionguard) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests>=2.27.1->predictionguard) (2.2.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests>=2.27.1->predictionguard) (2024.8.30)\n",
            "Downloading predictionguard-2.7.0-py2.py3-none-any.whl (21 kB)\n",
            "Installing collected packages: predictionguard\n",
            "Successfully installed predictionguard-2.7.0\n"
          ]
        }
      ],
      "source": [
        "! pip install predictionguard"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "from predictionguard import PredictionGuard\n",
        "from getpass import getpass"
      ],
      "metadata": {
        "id": "rOVhsPn42JEl"
      },
      "execution_count": 2,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "pg_access_token = getpass('Enter your Prediction Guard access api key: ')\n",
        "os.environ['PREDICTIONGUARD_API_KEY'] = pg_access_token"
      ],
      "metadata": {
        "id": "l8sDezef2Me8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "98ea8cf6-eff5-4355-a1e1-7cb29c49fc27"
      },
      "execution_count": 3,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your Prediction Guard access api key: ··········\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "client = PredictionGuard()"
      ],
      "metadata": {
        "id": "IG5p4NKiBK7Q"
      },
      "execution_count": 4,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# With and without prompt formatting"
      ],
      "metadata": {
        "id": "haoOqKSw2azm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "result = client.completions.create(\n",
        "    model=\"Hermes-3-Llama-3.1-8B\",\n",
        "    prompt=\"Tell me a joke\"\n",
        ")\n",
        "\n",
        "print(result['choices'][0]['text'])"
      ],
      "metadata": {
        "id": "9EBTZ6-V2dpo",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "c3bcbbc5-bf54-4411-ea54-617ff0accebf"
      },
      "execution_count": 11,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            " – I can’t remember any. I’m like a bank. I don’t keep deposits. Tell me a joke.\n",
            "I hate you – I was only kidding. I could never really hate you. I’m not capable of that emotion. Well, maybe in traffic but that doesn’t count.\n",
            "You said you’d never forget me. But you didn’t. Like I’m nothing. You just went ahead and forgot me.\n",
            "Where’s my damn coffee? I’m going to kill him. Wait,\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = \"\"\"<|im_start|>system\n",
        "{prompt}<|im_end|>\n",
        "<|im_start|>user\n",
        "Tell me a joke<|im_end|>\n",
        "<|im_start|>assistant<|im_end|>\"\"\"\n",
        "\n",
        "result = client.completions.create(\n",
        "    model=\"Hermes-3-Llama-3.1-8B\",\n",
        "    prompt=prompt\n",
        ")\n",
        "\n",
        "print(result['choices'][0]['text'])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QD47JPqy4m0a",
        "outputId": "bbedc4bf-dfa9-434c-e71c-ad8bd143b58d"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "Sure, here's a joke for you:\n",
            "\n",
            "Why don't scientists trust atoms? \n",
            "Because they make up everything!\n",
            "\n",
            "This joke plays on the dual meaning of \"make up\". In science, atoms truly do \"make up\" or constitute everything that exists. However, the phrase \"make up\" also means to fabricate or lie about something. So the joke humorously suggests that atoms are untrustworthy because they \"lie\" by making up everything. The juxtaposition of the two meanings is\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Automatic prompt formatting with a chat endpoing"
      ],
      "metadata": {
        "id": "oZvw-mv52TL6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "result = client.chat.completions.create(\n",
        "    model=\"Hermes-3-Llama-3.1-8B\",\n",
        "    messages=[{\"role\": \"user\", \"content\": \"Tell me a joke\"}]\n",
        ")\n",
        "\n",
        "print(result['choices'][0]['message']['content'])"
      ],
      "metadata": {
        "id": "lpBfvmQp2Ryu",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b955db36-6d3d-46d1-cdb9-cd7631f19189"
      },
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sure, here's a classic one for you:\n",
            "\n",
            "Why don't scientists trust atoms?\n",
            "\n",
            "Because they make up everything!\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "gP6yJn1IvTrb"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}