-
Notifications
You must be signed in to change notification settings - Fork 15.5k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Google Cloud Enterprise Search retriever (#7857)
Added a retriever that encapsulated Google Cloud Enterprise Search. --------- Co-authored-by: Bagatur <[email protected]>
- Loading branch information
Showing
4 changed files
with
467 additions
and
0 deletions.
There are no files selected for viewing
246 changes: 246 additions & 0 deletions
246
...tras/modules/data_connection/retrievers/integrations/google_cloud_enterprise_search.ipynb
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,246 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Google Cloud Enterprise Search\n", | ||
| "\n", | ||
| "\n", | ||
| "[Enterprise Search](https://cloud.google.com/enterprise-search) is a part of the Generative AI App Builder suite of tools offered by Google Cloud.\n", | ||
| "\n", | ||
| "Gen AI App Builder lets developers, even those with limited machine learning skills, quickly and easily tap into the power of Google’s foundation models, search expertise, and conversational AI technologies to create enterprise-grade generative AI applications. \n", | ||
| "\n", | ||
| "Enterprise Search lets organizations quickly build generative AI powered search engines for customers and employees.Enterprise Search is underpinned by a variety of Google Search technologies, including semantic search, which helps deliver more relevant results than traditional keyword-based search techniques by using natural language processing and machine learning techniques to infer relationships within the content and intent from the user’s query input. Enterprise Search also benefits from Google’s expertise in understanding how users search and factors in content relevance to order displayed results. \n", | ||
| "\n", | ||
| "Google Cloud offers Enterprise Search via Gen App Builder in Google Cloud Console and via an API for enterprise workflow integration. \n", | ||
| "\n", | ||
| "This notebook demonstrates how to configure Enterprise Search and use the Enterprise Search retriever. The Enterprise Search retriever encapsulates the [Generative AI App Builder Python client library](https://cloud.google.com/generative-ai-app-builder/docs/libraries#client-libraries-install-python) and uses it to access the Enterprise Search [Search Service API](https://cloud.google.com/python/docs/reference/discoveryengine/latest/google.cloud.discoveryengine_v1beta.services.search_service)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Install pre-requisites\n", | ||
| "\n", | ||
| "You need to install the `google-cloud-discoverengine` package to use the Enterprise Search retriever." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "! pip install google-cloud-discoveryengine" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Configure access to Google Cloud and Google Cloud Enterprise Search\n", | ||
| "\n", | ||
| "Enterprise Search is generally available for the allowlist (which means customers need to be approved for access) as of June 6, 2023. Contact your Google Cloud sales team for access and pricing details. We are previewing additional features that are coming soon to the generally available offering as part of our [Trusted Tester](https://cloud.google.com/ai/earlyaccess/join?hl=en) program. Sign up for [Trusted Tester](https://cloud.google.com/ai/earlyaccess/join?hl=en) and contact your Google Cloud sales team for an expedited trial.\n", | ||
| "\n", | ||
| "Before you can run this notebook you need to:\n", | ||
| "- Set or create a Google Cloud project and turn on Gen App Builder\n", | ||
| "- Create and populate an unstructured data store\n", | ||
| "- Set credentials to access `Enterprise Search API`" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Set or create a Google Cloud poject and turn on Gen App Builder\n", | ||
| "\n", | ||
| "Follow the instructions in the [Enterprise Search Getting Started guide](https://cloud.google.com/generative-ai-app-builder/docs/before-you-begin) to set/create a GCP project and enable Gen App Builder.\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Create and populate an unstructured data store\n", | ||
| "\n", | ||
| "[Use Google Cloud Console to create an unstructured data store](https://cloud.google.com/generative-ai-app-builder/docs/create-engine-es#unstructured-data) and populate it with the example PDF documents from the `gs://cloud-samples-data/gen-app-builder/search/alphabet-investor-pdfs` Cloud Storage folder. Make sure to use the `Cloud Storage (without metadata)` option." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Set credentials to access Enterprise Search API\n", | ||
| "\n", | ||
| "The [Gen App Builder client libraries](https://cloud.google.com/generative-ai-app-builder/docs/libraries) used by the Enterprise Search retriever provide high-level language support for authenticating to Gen App Builder programmatically. Client libraries support [Application Default Credentials (ADC)](https://cloud.google.com/docs/authentication/application-default-credentials); the libraries look for credentials in a set of defined locations and use those credentials to authenticate requests to the API. With ADC, you can make credentials available to your application in a variety of environments, such as local development or production, without needing to modify your application code.\n", | ||
| "\n", | ||
| "If running in [Google Colab](https://colab.google) authenticate with `google.colab.google.auth` otherwise follow one of the [supported methods](https://cloud.google.com/docs/authentication/application-default-credentials) to make sure that you Application Default Credentials are properly set." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import sys\n", | ||
| "\n", | ||
| "if \"google.colab\" in sys.modules:\n", | ||
| " from google.colab import auth as google_auth\n", | ||
| "\n", | ||
| " google_auth.authenticate_user()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "## Configure and use the Enterprise Search retriever\n", | ||
| "\n", | ||
| "The Enterprise Search retriever is implemented in the `langchain.retriever.GoogleCloudEntepriseSearchRetriever` class. The `get_relevan_documents` method returns a list of `langchain.schema.Document` documents where the `page_content` field of each document is populated with either an `extractive segment` or an `extractive answer` that matches a query. The `metadata` field is populated with metadata (if any) of a document from which the segments or answers were extracted.\n", | ||
| "\n", | ||
| "An extractive answer is verbatim text that is returned with each search result. It is extracted directly from the original document. Extractive answers are typically displayed near the top of web pages to provide an end user with a brief answer that is contextually relevant to their query. Extractive answers are available for website and unstructured search.\n", | ||
| "\n", | ||
| "An extractive segment is verbatim text that is returned with each search result. An extractive segment is usually more verbose than an extractive answer. Extractive segments can be displayed as an answer to a query, and can be used to perform post-processing tasks and as input for large language models to generate answers or new text. Extractive segments are available for unstructured search.\n", | ||
| "\n", | ||
| "For more information about extractive segments and extractive answers refer to [product documentation](https://cloud.google.com/generative-ai-app-builder/docs/snippets).\n", | ||
| "\n", | ||
| "When creating an instance of the retriever you can specify a number of parameters that control which Enterprise data store to access and how a natural language query is processed, including configurations for extractive answers and segments.\n", | ||
| "\n", | ||
| "The mandatory parameters are:\n", | ||
| "\n", | ||
| "- `project_id` - Your Google Cloud PROJECT_ID\n", | ||
| "- `search_engine_id` - The ID of the data store you want to use. \n", | ||
| "\n", | ||
| "The `project_id` and `search_engine_id` parameters can be provided explicitly in the retriever's constructor or through the environment variables - `PROJECT_ID` and `SEARCH_ENGINE_ID`.\n", | ||
| "\n", | ||
| "You can also configure a number of optional parameters, including:\n", | ||
| "\n", | ||
| "- `max_documents` - The maximum number of documents used to provide extractive segments or extractive answers\n", | ||
| "- `get_extractive_answers` - By default, the retriever is configured to return extractive segments. Set this field to `True` to return extractive answers\n", | ||
| "- `max_extractive_answer_count` - The maximum number of extractive answers returned in each search result.\n", | ||
| " At most 5 answers will be returned\n", | ||
| "- `max_extractive_segment_count` - The maximum number of extractive segments returned in each search result.\n", | ||
| " Currently one segment will be returned\n", | ||
| "- `filter` - The filter expression that allows you filter the search results based on the metadata associated with the documents in the searched data store. \n", | ||
| "- `query_expansion_condition` - Specification to determine under which conditions query expansion should occur.\n", | ||
| " 0 - Unspecified query expansion condition. In this case, server behavior defaults to disabled.\n", | ||
| " 1 - Disabled query expansion. Only the exact search query is used, even if SearchResponse.total_size is zero.\n", | ||
| " 2 - Automatic query expansion built by the Search API.\n", | ||
| "\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Configure and use the retriever with extractve segments" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "from langchain.retrievers import GoogleCloudEnterpriseSearchRetriever\n", | ||
| "\n", | ||
| "PROJECT_ID = \"<YOUR PROJECT ID>\" # Set to your Project ID\n", | ||
| "SEARCH_ENGINE_ID = \"<YOUR SEARCH ENGINE ID>\" # Set to your data store ID" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "retriever = GoogleCloudEnterpriseSearchRetriever(\n", | ||
| " project_id=PROJECT_ID,\n", | ||
| " search_engine_id=SEARCH_ENGINE_ID,\n", | ||
| " max_documents=3,\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "query = \"What are Alphabet's Other Bets?\"\n", | ||
| "\n", | ||
| "result = retriever.get_relevant_documents(query)\n", | ||
| "for doc in result:\n", | ||
| " print(doc)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "### Configure and use the retriever with extractve answers " | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "retriever = GoogleCloudEnterpriseSearchRetriever(\n", | ||
| " project_id=PROJECT_ID,\n", | ||
| " search_engine_id=SEARCH_ENGINE_ID,\n", | ||
| " max_documents=3,\n", | ||
| " max_extractive_answer_count=3,\n", | ||
| " get_extractive_answers=True,\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "query = \"What are Alphabet's Other Bets?\"\n", | ||
| "\n", | ||
| "result = retriever.get_relevant_documents(query)\n", | ||
| "for doc in result:\n", | ||
| " print(doc)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "base", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.10.10" | ||
| }, | ||
| "orig_nbformat": 4 | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.