-
Notifications
You must be signed in to change notification settings - Fork 894
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into release/promptflow/1.15.0.post1
- Loading branch information
Showing
138 changed files
with
1,505,041 additions
and
2,064 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # Rerank | ||
|
|
||
| ## Introduction | ||
| Rerank is a semantic search tool that improves search quality with a semantic-based reranking system which can contextualize the meaning of a user's query beyond keyword relevance. This tool works best with look up tool as a ranker after the initial retrieval. The list of current supported ranking method is as follows. | ||
|
|
||
| | Name | Description | | ||
| | --- | --- | | ||
| | BM25 | BM25 is an open source ranking algorithm to measure the relevance of documents to a given query | | ||
| | Scaled Score Fusion | Scaled Score Fusion calculates a scaled relevance score. | | ||
| | Cohere Rerank | Cohere Rerank is the market’s leading reranking model used for semantic search and retrieval-augmented generation (RAG). | | ||
|
|
||
| ## Requirements | ||
| - For AzureML users, the tool is installed in default image, you can use the tool without extra installation. | ||
| - For local users, | ||
|
|
||
| `pip install promptflow-vectordb` | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| BM25 and Scaled Score Fusion are included as default reranking methods. To use cohere rerank model, you should create serverless deployment to the model, and establish connection between the tool and the resource as follows. | ||
|
|
||
| - Add Serverless Model connection. Fill "API base" and "API key" field to your serverless deployment. | ||
|
|
||
|
|
||
| ## Inputs | ||
|
|
||
| | Name | Type | Description | Required | | ||
| |------------------------|-------------|-----------------------------------------------------------------------|----------| | ||
| | query | string | the question relevant to your input documents | Yes | | ||
| | ranker_parameters | string | the type of ranking methods to use | Yes | | ||
| | result_groups | object | the list of document chunks to rerank. Normally this is output from lookup | Yes | | ||
| | top_k | int | the maximum number of relevant documents to return | No | | ||
|
|
||
|
|
||
|
|
||
| ## Outputs | ||
|
|
||
| | Return Type | Description | | ||
| |-------------|------------------------------------------| | ||
| | text | text of the entity | | ||
| | metadata | metadata like file path and url | | ||
| | additional_fields | metadata and rerank score | | ||
|
|
||
| <details> | ||
| <summary>Output</summary> | ||
|
|
||
| ```json | ||
| [ | ||
| { | ||
| "text": "sample text", | ||
| "metadata": | ||
| { | ||
| "filepath": "sample_file_path", | ||
| "metadata_json_string": "meta_json_string" | ||
| "title": "", | ||
| "url": "" | ||
| }, | ||
| "additional_fields": | ||
| { | ||
| "filepath": "sample_file_path", | ||
| "metadata_json_string": "meta_json_string" | ||
| "title": "", | ||
| "url": "", | ||
| "@promptflow_vectordb.reranker_score": 0.013795365 | ||
| } | ||
| } | ||
| ] | ||
| ``` | ||
| </details> |
69 changes: 69 additions & 0 deletions
69
examples/flows/integrations/continuous-monitoring-with-pipeline/README.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # Continuous Monitoring Pipeline | ||
|
|
||
| This tutorial describes an advanced use case of [running flows in Azure ML Pipelines](https://github.com/microsoft/promptflow/blob/main/examples/tutorials/run-flow-with-pipeline/pipeline.ipynb). | ||
| The detailed explanations of the prerequisites and principles can be found in the aforementioned article. | ||
| Continuous monitoring is necessary to maintain the quality, performance and efficiency of Generative AI applications. | ||
| These factors directly impact the user experience and operational costs. | ||
|
|
||
| We will run evaluations on a basic chatbot flow, then aggregate the results to export and visualize the metrics. | ||
| The flows used in this pipeline are described below: | ||
| - [Basic Chat](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-basic) | ||
| - [Q&A Evaluation](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-qna-rag-metrics) | ||
| - [Perceived Intelligence Evaluation](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-perceived-intelligence) | ||
| - [Summarization Evaluation](https://github.com/microsoft/promptflow/tree/main/examples/flows/evaluation/eval-summarization) | ||
|
|
||
| Connections used in this flow: | ||
| - `azure_open_ai_connection` connection (Azure OpenAI). | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| ### Prompt flow SDK: | ||
| - Azure cloud setup: | ||
| - An Azure account with an active subscription - [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) | ||
| - Create an Azure ML resource from Azure portal - [Create a Azure ML workspace](https://ms.portal.azure.com/#view/Microsoft_Azure_Marketplace/MarketplaceOffersBlade/searchQuery/machine%20learning) | ||
| - Connect to your workspace then setup a basic computer cluster - [Configure workspace](https://github.com/microsoft/promptflow/blob/main/examples/configuration.ipynb) | ||
| - Local environment setup: | ||
| - A python environment | ||
| - Installed Azure Machine Learning Python SDK v2 - [install instructions](https://github.com/microsoft/promptflow/blob/main/examples/README.md) - check the getting started section and make sure version of 'azure-ai-ml' is higher than `1.12.0` | ||
|
|
||
| Note: when using the Prompt flow SDK, it may be useful to also install the [`Prompt flow for VS Code`](https://marketplace.visualstudio.com/items?itemName=prompt-flow.prompt-flow) extension (if using VS Code). | ||
|
|
||
| ### Azure AI/ML Studio: | ||
| Start a compute session. | ||
| The follow the installation steps described in the notebook. | ||
|
|
||
| ## Setup connections | ||
| Ensure that you have a connection to Azure OpenAI with the following deployments: | ||
| - `gpt-35-turbo` | ||
| - `gpt-4` | ||
|
|
||
| ## Run pipeline | ||
|
|
||
| Run the notebook's steps until `3.2.2 Submit the job` to start the pipeline in Azure ML Studio. | ||
|
|
||
| ## Pipeline description | ||
| The first node reads the evaluation dataset. | ||
| The second node is the main flow that will be monitored, it takes the output of the evaluation dataset as a `data` input. | ||
| After the main flow's node has completed, its output will go to 3 nodes: | ||
| - Q&A Evaluation | ||
| - Perceived Intelligence Evaluation | ||
| - Simple Summarization | ||
|
|
||
| The Simple Summarization and the main nodes' outputs will become the Summarization Evaluation node's input. | ||
|
|
||
| Finally, all the evaluation metrics will be aggregated and displayed in Azure ML Pipeline's interface. | ||
|
|
||
|  | ||
|
|
||
| ## Metrics visualization | ||
| In the node `Convert evaluation results to parquet` Metrics tab, the aggregated metrics will be displayed. | ||
|
|
||
|  | ||
|
|
||
| The evolution of the metrics can be monitored by comparing multiple pipeline runs: | ||
|
|
||
|  | ||
|
|
||
|  | ||
| ## Contact | ||
| Please reach out to Lou Bigard (<[email protected]>) with any issues. |
45 changes: 45 additions & 0 deletions
45
...ons/continuous-monitoring-with-pipeline/flows/standard/simple-summarization/flow.dag.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| $schema: https://azuremlschemas.azureedge.net/promptflow/latest/Flow.schema.json | ||
| environment: | ||
| python_requirements_txt: requirements.txt | ||
| inputs: | ||
| answer: | ||
| type: string | ||
| outputs: | ||
| summary: | ||
| type: string | ||
| reference: ${summarize_text_content.output} | ||
| nodes: | ||
| - name: summarize_text_content | ||
| use_variants: true | ||
| node_variants: | ||
| summarize_text_content: | ||
| default_variant_id: variant_0 | ||
| variants: | ||
| variant_0: | ||
| node: | ||
| type: llm | ||
| source: | ||
| type: code | ||
| path: summarize_text_content.jinja2 | ||
| inputs: | ||
| deployment_name: gpt-35-turbo | ||
| model: gpt-3.5-turbo | ||
| max_tokens: 128 | ||
| temperature: 0.2 | ||
| text: ${inputs.answer} | ||
| connection: open_ai_connection | ||
| api: chat | ||
| variant_1: | ||
| node: | ||
| type: llm | ||
| source: | ||
| type: code | ||
| path: summarize_text_content__variant_1.jinja2 | ||
| inputs: | ||
| deployment_name: gpt-35-turbo | ||
| model: gpt-3.5-turbo | ||
| max_tokens: 256 | ||
| temperature: 0.3 | ||
| text: ${inputs.answer} | ||
| connection: open_ai_connection | ||
| api: chat |
2 changes: 2 additions & 0 deletions
2
.../continuous-monitoring-with-pipeline/flows/standard/simple-summarization/requirements.txt
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| promptflow[azure]>=1.7.0 | ||
| promptflow-tools |
7 changes: 7 additions & 0 deletions
7
...onitoring-with-pipeline/flows/standard/simple-summarization/summarize_text_content.jinja2
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # system: | ||
| Please summarize the following text in one paragraph. 100 words. | ||
| Do not add any information that is not in the text. | ||
|
|
||
| # user: | ||
| Text: {{text}} | ||
| Summary: |
Oops, something went wrong.