-
Notifications
You must be signed in to change notification settings - Fork 15.9k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
We are introducing the py integration to Javelin AI Gateway www.getjavelin.io. Javelin is an enterprise-scale fast llm router & gateway. Could you please review and let us know if there is anything missing. Javelin AI Gateway wraps Embedding, Chat and Completion LLMs. Uses javelin_sdk under the covers (pip install javelin_sdk). Author: Sharath Rajasekar, Twitter: @sharathr, @JavelinAI Thanks!!
- Loading branch information
Showing
9 changed files
with
837 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,242 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "62bacc68-1976-44eb-9316-d5baf54bf595", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Javelin AI Gateway Tutorial\n", | ||
| "\n", | ||
| "This Jupyter Notebook will explore how to interact with the Javelin AI Gateway using the Python SDK. \n", | ||
| "The Javelin AI Gateway facilitates the utilization of large language models (LLMs) like OpenAI, Cohere, Anthropic, and others by \n", | ||
| "providing a secure and unified endpoint. The gateway itself provides a centralized mechanism to roll out models systematically, \n", | ||
| "provide access security, policy & cost guardrails for enterprises, etc., \n", | ||
| "\n", | ||
| "For a complete listing of all the features & benefits of Javelin, please visit www.getjavelin.io\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "e52185f8-132b-4585-b73d-6fee928ac199", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Step 1: Introduction\n", | ||
| "[The Javelin AI Gateway](https://www.getjavelin.io) is an enterprise-grade API Gateway for AI applications. It integrates robust access security, ensuring secure interactions with large language models. Learn more in the [official documentation](https://docs.getjavelin.io).\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "2e2acdb3-e3b8-422b-b077-7a0d63d18349", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Step 2: Installation\n", | ||
| "Before we begin, we must install the `javelin_sdk` and set up the Javelin API key as an environment variable. " | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 5, | ||
| "id": "e91518a4-43ce-443e-b4c0-dbc652eb749f", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stdout", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "Requirement already satisfied: javelin_sdk in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (0.1.8)\n", | ||
| "Requirement already satisfied: httpx<0.25.0,>=0.24.0 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from javelin_sdk) (0.24.1)\n", | ||
| "Requirement already satisfied: pydantic<2.0.0,>=1.10.7 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from javelin_sdk) (1.10.12)\n", | ||
| "Requirement already satisfied: certifi in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpx<0.25.0,>=0.24.0->javelin_sdk) (2023.5.7)\n", | ||
| "Requirement already satisfied: httpcore<0.18.0,>=0.15.0 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpx<0.25.0,>=0.24.0->javelin_sdk) (0.17.3)\n", | ||
| "Requirement already satisfied: idna in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpx<0.25.0,>=0.24.0->javelin_sdk) (3.4)\n", | ||
| "Requirement already satisfied: sniffio in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpx<0.25.0,>=0.24.0->javelin_sdk) (1.3.0)\n", | ||
| "Requirement already satisfied: typing-extensions>=4.2.0 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from pydantic<2.0.0,>=1.10.7->javelin_sdk) (4.7.1)\n", | ||
| "Requirement already satisfied: h11<0.15,>=0.13 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpcore<0.18.0,>=0.15.0->httpx<0.25.0,>=0.24.0->javelin_sdk) (0.14.0)\n", | ||
| "Requirement already satisfied: anyio<5.0,>=3.0 in /usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages (from httpcore<0.18.0,>=0.15.0->httpx<0.25.0,>=0.24.0->javelin_sdk) (3.7.1)\n", | ||
| "Note: you may need to restart the kernel to use updated packages.\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "pip install 'javelin_sdk'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "53b546dc-9ca3-4602-9a7b-d733d99e8e2f", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Step 3: Completions Example\n", | ||
| "This section will demonstrate how to interact with the Javelin AI Gateway to get completions from a large language model. Here is a Python script that demonstrates this:\n", | ||
| "(note) assumes that you have setup a route in the gateway called 'eng_dept03'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 6, | ||
| "id": "d36949f0-5354-44ca-9a31-70c769344319", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "ename": "ImportError", | ||
| "evalue": "cannot import name 'JavelinAIGateway' from 'langchain.llms' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/llms/__init__.py)", | ||
| "output_type": "error", | ||
| "traceback": [ | ||
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | ||
| "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", | ||
| "Cell \u001b[0;32mIn[6], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchains\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m LLMChain\n\u001b[0;32m----> 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mllms\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m JavelinAIGateway\n\u001b[1;32m 3\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mprompts\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m PromptTemplate\n\u001b[1;32m 5\u001b[0m route_completions \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124meng_dept03\u001b[39m\u001b[38;5;124m\"\u001b[39m\n", | ||
| "\u001b[0;31mImportError\u001b[0m: cannot import name 'JavelinAIGateway' from 'langchain.llms' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/llms/__init__.py)" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain.chains import LLMChain\n", | ||
| "from langchain.llms import JavelinAIGateway\n", | ||
| "from langchain.prompts import PromptTemplate\n", | ||
| "\n", | ||
| "route_completions = \"eng_dept03\"\n", | ||
| "\n", | ||
| "gateway = JavelinAIGateway(\n", | ||
| " gateway_uri=\"http://localhost:8000\", # replace with service URL or host/port of Javelin\n", | ||
| " route=route_completions,\n", | ||
| " model_name=\"text-davinci-003\",\n", | ||
| ")\n", | ||
| "\n", | ||
| "prompt = PromptTemplate(\"Translate the following English text to French: {text}\")\n", | ||
| "\n", | ||
| "llmchain = LLMChain(llm=gateway, prompt=prompt)\n", | ||
| "result = llmchain.run(\"podcast player\")\n", | ||
| "\n", | ||
| "print(result)\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "6b63fe93-2e77-4ea9-b8e7-dec2b96b8e95", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Step 4: Embeddings Example\n", | ||
| "This section demonstrates how to use the Javelin AI Gateway to obtain embeddings for text queries and documents. Here is a Python script that illustrates this:\n", | ||
| "(note) assumes that you have setup a route in the gateway called 'embeddings'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 9, | ||
| "id": "878e6c1d-be7f-49de-825c-43c266c8714e", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "ename": "ImportError", | ||
| "evalue": "cannot import name 'JavelinAIGatewayEmbeddings' from 'langchain.embeddings' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/embeddings/__init__.py)", | ||
| "output_type": "error", | ||
| "traceback": [ | ||
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | ||
| "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", | ||
| "Cell \u001b[0;32mIn[9], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01membeddings\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m JavelinAIGatewayEmbeddings\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01membeddings\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mopenai\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m OpenAIEmbeddings\n\u001b[1;32m 4\u001b[0m embeddings \u001b[38;5;241m=\u001b[39m JavelinAIGatewayEmbeddings(\n\u001b[1;32m 5\u001b[0m gateway_uri\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mhttp://localhost:8000\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;66;03m# replace with service URL or host/port of Javelin\u001b[39;00m\n\u001b[1;32m 6\u001b[0m route\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124membeddings\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[1;32m 7\u001b[0m )\n", | ||
| "\u001b[0;31mImportError\u001b[0m: cannot import name 'JavelinAIGatewayEmbeddings' from 'langchain.embeddings' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/embeddings/__init__.py)" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain.embeddings import JavelinAIGatewayEmbeddings\n", | ||
| "from langchain.embeddings.openai import OpenAIEmbeddings\n", | ||
| "\n", | ||
| "embeddings = JavelinAIGatewayEmbeddings(\n", | ||
| " gateway_uri=\"http://localhost:8000\", # replace with service URL or host/port of Javelin\n", | ||
| " route=\"embeddings\",\n", | ||
| ")\n", | ||
| "\n", | ||
| "print(embeddings.embed_query(\"hello\"))\n", | ||
| "print(embeddings.embed_documents([\"hello\"]))\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "07c6691b-d333-4598-b2b7-c0933ed75937", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Step 5: Chat Example\n", | ||
| "This section illustrates how to interact with the Javelin AI Gateway to facilitate a chat with a large language model. Here is a Python script that demonstrates this:\n", | ||
| "(note) assumes that you have setup a route in the gateway called 'mychatbot_route'" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 8, | ||
| "id": "653ef88c-36cd-4730-9c12-43c246b551f1", | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "ename": "ImportError", | ||
| "evalue": "cannot import name 'ChatJavelinAIGateway' from 'langchain.chat_models' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/chat_models/__init__.py)", | ||
| "output_type": "error", | ||
| "traceback": [ | ||
| "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", | ||
| "\u001b[0;31mImportError\u001b[0m Traceback (most recent call last)", | ||
| "Cell \u001b[0;32mIn[8], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mchat_models\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m ChatJavelinAIGateway\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mlangchain\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mschema\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m HumanMessage, SystemMessage\n\u001b[1;32m 4\u001b[0m messages \u001b[38;5;241m=\u001b[39m [\n\u001b[1;32m 5\u001b[0m SystemMessage(\n\u001b[1;32m 6\u001b[0m content\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mYou are a helpful assistant that translates English to French.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m (...)\u001b[0m\n\u001b[1;32m 10\u001b[0m ),\n\u001b[1;32m 11\u001b[0m ]\n", | ||
| "\u001b[0;31mImportError\u001b[0m: cannot import name 'ChatJavelinAIGateway' from 'langchain.chat_models' (/usr/local/Caskroom/miniconda/base/lib/python3.11/site-packages/langchain/chat_models/__init__.py)" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "from langchain.chat_models import ChatJavelinAIGateway\n", | ||
| "from langchain.schema import HumanMessage, SystemMessage\n", | ||
| "\n", | ||
| "messages = [\n", | ||
| " SystemMessage(\n", | ||
| " content=\"You are a helpful assistant that translates English to French.\"\n", | ||
| " ),\n", | ||
| " HumanMessage(\n", | ||
| " content=\"Artificial Intelligence has the power to transform humanity and make the world a better place\"\n", | ||
| " ),\n", | ||
| "]\n", | ||
| "\n", | ||
| "chat = ChatJavelinAIGateway(\n", | ||
| " gateway_uri=\"http://localhost:8000\", # replace with service URL or host/port of Javelin\n", | ||
| " route=\"mychatbot_route\",\n", | ||
| " model_name=\"gpt-3.5-turbo\",\n", | ||
| " params={\n", | ||
| " \"temperature\": 0.1\n", | ||
| " }\n", | ||
| ")\n", | ||
| "\n", | ||
| "print(chat(messages))\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "id": "6eb9cf33-6505-4e05-808b-645856463a8e", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Step 6: Conclusion\n", | ||
| "This tutorial introduced the Javelin AI Gateway and demonstrated how to interact with it using the Python SDK. \n", | ||
| "Remember to check the Javelin [Python SDK](https://www.github.com/getjavelin.io/javelin-python) for more examples and to explore the official documentation for additional details.\n", | ||
| "\n", | ||
| "Happy coding!" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "Python 3 (ipykernel)", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.4" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 5 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| # Javelin AI Gateway | ||
|
|
||
| [The Javelin AI Gateway](https://www.getjavelin.io) service is a high-performance, enterprise grade API Gateway for AI applications. | ||
| It is designed to streamline the usage and access of various large language model (LLM) providers, | ||
| such as OpenAI, Cohere, Anthropic and custom large language models within an organization by incorporating | ||
| robust access security for all interactions with LLMs. | ||
|
|
||
| Javelin offers a high-level interface that simplifies the interaction with LLMs by providing a unified endpoint | ||
| to handle specific LLM related requests. | ||
|
|
||
| See the Javelin AI Gateway [documentation](https://docs.getjavelin.io) for more details. | ||
| [Javelin Python SDK](https://www.github.com/getjavelin/javelin-python) is an easy to use client library meant to be embedded into AI Applications | ||
|
|
||
| ## Installation and Setup | ||
|
|
||
| Install `javelin_sdk` to interact with Javelin AI Gateway: | ||
|
|
||
| ```sh | ||
| pip install 'javelin_sdk' | ||
| ``` | ||
|
|
||
| Set the Javelin's API key as an environment variable: | ||
|
|
||
| ```sh | ||
| export JAVELIN_API_KEY=... | ||
| ``` | ||
|
|
||
| ## Completions Example | ||
|
|
||
| ```python | ||
|
|
||
| from langchain.chains import LLMChain | ||
| from langchain.llms import JavelinAIGateway | ||
| from langchain.prompts import PromptTemplate | ||
|
|
||
| route_completions = "eng_dept03" | ||
|
|
||
| gateway = JavelinAIGateway( | ||
| gateway_uri="http://localhost:8000", | ||
| route=route_completions, | ||
| model_name="text-davinci-003", | ||
| ) | ||
|
|
||
| llmchain = LLMChain(llm=gateway, prompt=prompt) | ||
| result = llmchain.run("podcast player") | ||
|
|
||
| print(result) | ||
|
|
||
| ``` | ||
|
|
||
| ## Embeddings Example | ||
|
|
||
| ```python | ||
| from langchain.embeddings import JavelinAIGatewayEmbeddings | ||
| from langchain.embeddings.openai import OpenAIEmbeddings | ||
|
|
||
| embeddings = JavelinAIGatewayEmbeddings( | ||
| gateway_uri="http://localhost:8000", | ||
| route="embeddings", | ||
| ) | ||
|
|
||
| print(embeddings.embed_query("hello")) | ||
| print(embeddings.embed_documents(["hello"])) | ||
| ``` | ||
|
|
||
| ## Chat Example | ||
| ```python | ||
| from langchain.chat_models import ChatJavelinAIGateway | ||
| from langchain.schema import HumanMessage, SystemMessage | ||
|
|
||
| messages = [ | ||
| SystemMessage( | ||
| content="You are a helpful assistant that translates English to French." | ||
| ), | ||
| HumanMessage( | ||
| content="Artificial Intelligence has the power to transform humanity and make the world a better place" | ||
| ), | ||
| ] | ||
|
|
||
| chat = ChatJavelinAIGateway( | ||
| gateway_uri="http://localhost:8000", | ||
| route="mychatbot_route", | ||
| model_name="gpt-3.5-turbo" | ||
| params={ | ||
| "temperature": 0.1 | ||
| } | ||
| ) | ||
|
|
||
| print(chat(messages)) | ||
|
|
||
| ``` | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.