-
Notifications
You must be signed in to change notification settings - Fork 238
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Add documentation for DSPY integration --------- Co-authored-by: Boris Feld <[email protected]>
- Loading branch information
1 parent
c97792e
commit 7da0949
Showing
13 changed files
with
278 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
159 changes: 159 additions & 0 deletions
159
apps/opik-documentation/documentation/docs/cookbook/dspy.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "---\n", | ||
| "description: Cookbook that showcases Opik's integration with DSPy\n", | ||
| "---\n", | ||
| "\n", | ||
| "# Using Opik with DSPy\n", | ||
| "\n", | ||
| "[DSPy](https://dspy.ai/) is the framework for programming—rather than prompting—language models.\n", | ||
| "\n", | ||
| "In this guide, we will showcase how to integrate Opik with DSPy so that all the DSPy calls are logged as traces in Opik." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Creating an account on Comet.com\n", | ||
| "\n", | ||
| "[Comet](https://www.comet.com/site?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) provides a hosted version of the Opik platform, [simply create an account](https://www.comet.com/signup?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) and grab you API Key.\n", | ||
| "\n", | ||
| "> You can also run the Opik platform locally, see the [installation guide](https://www.comet.com/docs/opik/self-host/overview/?from=llm&utm_source=opik&utm_medium=colab&utm_content=dspy&utm_campaign=opik) for more information." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "%pip install --upgrade opik dspy" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 1, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "OPIK: Opik is already configured. You can check the settings by viewing the config file at /Users/jacquesverre/.opik.config\n" | ||
| ] | ||
| } | ||
| ], | ||
| "source": [ | ||
| "import opik\n", | ||
| "\n", | ||
| "opik.configure(use_local=False)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 2, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import os\n", | ||
| "import getpass\n", | ||
| "\n", | ||
| "if \"OPENAI_API_KEY\" not in os.environ:\n", | ||
| " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter your OpenAI API key: \")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Logging traces\n", | ||
| "\n", | ||
| "In order to log traces to Opik, you will need to set the `opik` callback:" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 3, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import dspy\n", | ||
| "from opik.integrations.dspy.callback import OpikCallback\n", | ||
| "\n", | ||
| "lm = dspy.LM(\"openai/gpt-4o-mini\")\n", | ||
| "\n", | ||
| "project_name = \"DSPY\"\n", | ||
| "opik_callback = OpikCallback(project_name=project_name)\n", | ||
| "\n", | ||
| "dspy.configure(lm=lm, callbacks=[opik_callback])" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 4, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "name": "stderr", | ||
| "output_type": "stream", | ||
| "text": [ | ||
| "WARNING:langfuse:Langfuse client is disabled since no public_key was provided as a parameter or environment variable 'LANGFUSE_PUBLIC_KEY'. See our docs: https://langfuse.com/docs/sdk/python/low-level-sdk#initialize-client\n", | ||
| "OPIK: Started logging traces to the \"DSPY\" project at https://www.comet.com/opik/jacques-comet/redirect/projects?name=DSPY.\n" | ||
| ] | ||
| }, | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "Prediction(\n", | ||
| " reasoning='The meaning of life is a philosophical question that has been contemplated by humans for centuries. Different cultures, religions, and individuals have proposed various interpretations. Some suggest that the meaning of life is to seek happiness, fulfillment, and personal growth, while others believe it is about serving a higher purpose or contributing to the well-being of others. Ultimately, the meaning of life may vary from person to person, shaped by personal experiences, beliefs, and values.',\n", | ||
| " answer=\"The meaning of life is subjective and can vary greatly among individuals. It may involve seeking happiness, personal growth, and contributing to the well-being of others, or fulfilling a higher purpose, depending on one's beliefs and experiences.\"\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| "execution_count": 4, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "cot = dspy.ChainOfThought(\"question -> answer\")\n", | ||
| "cot(question=\"What is the meaning of life?\")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "The trace is now logged to the Opik platform:\n", | ||
| "\n", | ||
| "" | ||
| ] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "py312_llm_eval", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.12.4" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 4 | ||
| } |
60 changes: 60 additions & 0 deletions
60
apps/opik-documentation/documentation/docs/tracing/integrations/dspy.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| --- | ||
| sidebar_label: DSPy | ||
| description: Describes how to track DSPy calls using Opik | ||
| --- | ||
|
|
||
| # DSPy | ||
|
|
||
| [DSPy](https://dspy.ai/) is the framework for programming—rather than prompting—language models. | ||
|
|
||
| Opik integrates with DSPy to log traces for all DSPy calls. | ||
|
|
||
| <div style="display: flex; align-items: center; flex-wrap: wrap; margin: 20px 0;"> | ||
| <span style="margin-right: 10px;">You can check out the Colab Notebook if you'd like to jump straight to the code:</span> | ||
| <a href="https://colab.research.google.com/github/comet-ml/opik/blob/main/apps/opik-documentation/documentation/docs/cookbook/dspy.ipynb" target="_blank" rel="noopener noreferrer"> | ||
| <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab" style="vertical-align: middle;"/> | ||
| </a> | ||
| </div> | ||
|
|
||
| ## Getting started | ||
|
|
||
| First, ensure you have both `opik` and `dspy` installed: | ||
|
|
||
| ```bash | ||
| pip install opik dspy | ||
| ``` | ||
|
|
||
| In addition, you can configure Opik using the `opik configure` command which will prompt you for the correct local server address or if you are using the Cloud platfrom your API key: | ||
|
|
||
| ```bash | ||
| opik configure | ||
| ``` | ||
|
|
||
| ## Logging DSPy calls | ||
|
|
||
| To log a DSPy pipeline run, you can use the [`OpikCallback`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/dspy/OpikCallback.html). This callback will log each DSPy run to Opik: | ||
|
|
||
| ```python | ||
| import dspy | ||
| from opik.integrations.dspy.callback import OpikCallback | ||
|
|
||
| project_name = "DSPY" | ||
|
|
||
| lm = dspy.LM( | ||
| model="openai/gpt-4o-mini", | ||
| ) | ||
| dspy.configure(lm=lm) | ||
|
|
||
|
|
||
| opik_callback = OpikCallback(project_name=project_name) | ||
| dspy.settings.configure( | ||
| callbacks=[opik_callback], | ||
| ) | ||
|
|
||
| cot = dspy.ChainOfThought("question -> answer") | ||
| cot(question="What is the meaning of life?") | ||
| ``` | ||
|
|
||
| Each run will now be logged to the Opik platform: | ||
|
|
||
|  |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file added
BIN
+570 KB
apps/opik-documentation/documentation/static/img/cookbook/dspy_trace_cookbook.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,3 +10,5 @@ langchain_core | |
| botocore | ||
| anthropic | ||
| haystack-ai | ||
| dspy-ai | ||
| cloudpickle | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 changes: 5 additions & 0 deletions
5
apps/opik-documentation/python-sdk-docs/source/integrations/dspy/OpikCallback.rst
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| OpikCallback | ||
| ============ | ||
|
|
||
| .. autoclass:: opik.integrations.dspy.OpikCallback | ||
| :members: |
Oops, something went wrong.