[Feature] Add Retrieval Augmented Generation search processors #1275

austintlee · 2023-09-01T21:02:33Z

Description

Issues Resolved

[List any issues this PR will resolve]

Check List

[ x] New functionality includes testing.
- [x ] All tests pass
[ x] New functionality has been documented.
- [ x] New functionality has javadoc added
[ x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Austin Lee <[email protected]>

austintlee · 2023-09-01T21:05:20Z

curl -X PUT http://localhost:9200/_cluster/settings -H "Content-Type: application/json" \
-d '{"persistent": {"plugins.ml_commons.rag_pipeline_feature_enabled" : true}}' | jq .
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   190  100   116  100    74   1830   1168 --:--:-- --:--:-- --:--:--  3454
{
  "acknowledged": true,
  "persistent": {
    "plugins": {
      "ml_commons": {
        "rag_pipeline_feature_enabled": "true"
      }
    }
  },
  "transient": {}
}

[2023-09-01T13:59:54,088][INFO ][o.o.c.s.ClusterSettings  ] [integTest-0] updating [plugins.ml_commons.rag_pipeline_feature_enabled] from [false] to [true]

austintlee · 2023-09-01T21:05:39Z

@ylwu-amzn

Signed-off-by: Austin Lee <[email protected]>

codecov · 2023-09-01T21:35:03Z

Codecov Report

Merging #1275 (8f4d90d) into feature/conversation (da7bdc9) will increase coverage by 0.34%.
Report is 1 commits behind head on feature/conversation.
The diff coverage is 91.78%.

❗ Current head 8f4d90d differs from pull request most recent head 739d96a. Consider uploading reports for the commit 739d96a to get more accurate results

@@                    Coverage Diff                     @@
##             feature/conversation    #1275      +/-   ##
==========================================================
+ Coverage                   79.04%   79.39%   +0.34%     
- Complexity                   2148     2224      +76     
==========================================================
  Files                         172      187      +15     
  Lines                        8673     8952     +279     
  Branches                      870      893      +23     
==========================================================
+ Hits                         6856     7107     +251     
- Misses                       1422     1441      +19     
- Partials                      395      404       +9

Flag	Coverage Δ
ml-commons	`79.39% <91.78%> (+0.34%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...ing/generative/GenerativeQAProcessorConstants.java	`0.00% <0.00%> (ø)`
...rg/opensearch/ml/plugin/MachineLearningPlugin.java	`92.62% <50.00%> (-6.45%)`	⬇️
...nerative/client/MachineLearningInternalClient.java	`72.41% <72.41%> (ø)`
...nswering/generative/ext/GenerativeQAParamUtil.java	`88.88% <88.88%> (ø)`
...ring/generative/GenerativeQAResponseProcessor.java	`89.79% <89.79%> (ø)`
.../org/opensearch/ml/settings/MLCommonsSettings.java	`100.00% <100.00%> (ø)`
...ering/generative/GenerativeQARequestProcessor.java	`100.00% <100.00%> (ø)`
...answering/generative/GenerativeSearchResponse.java	`100.00% <100.00%> (ø)`
.../generative/client/ConversationalMemoryClient.java	`100.00% <100.00%> (ø)`
...ng/generative/ext/GenerativeQAParamExtBuilder.java	`100.00% <100.00%> (ø)`
... and 7 more

... and 2 files with indirect coverage changes

Signed-off-by: Austin Lee <[email protected]>

austintlee · 2023-09-04T17:32:53Z

The failing test(s) seem to be flaky ones. I ran the last one that failed on my machine and it passed.

dhrubo-os · 2023-09-05T17:54:34Z

The failing test(s) seem to be flaky ones. I ran the last one that failed on my machine and it passed.

Seems like it's a jarhell issue?

Task :opensearch-ml-plugin:jarHell FAILED

dhrubo-os · 2023-09-05T17:25:19Z

plugin/src/main/java/org/opensearch/ml/settings/MLCommonsSettings.java

@@ -130,4 +130,7 @@ private MLCommonsSettings() {}
        );

    public static final Setting<Boolean> ML_COMMONS_MEMORY_FEATURE_ENABLED = ConversationalIndexConstants.ML_COMMONS_MEMORY_FEATURE_ENABLED;
+    // Feature flag for enabling search processors for Retrieval Augmented Generation using OpenSearch and Remote Inference.
+    public static final Setting<Boolean> ML_COMMONS_RAG_PIPELINE_FEATURE_ENABLED = Setting


I'm curious to know what would actually happen if someone were to enable the ML_COMMONS_RAG_PIPELINE_FEATURE_ENABLED flag but not enable the ML_COMMONS_MEMORY_FEATURE_ENABLED flag, or vice versa?

Pipeline, no Memory: you can use the pipeline without memory anyway. Just a straightforward RAG pipeline without any conversation history. Trying to use memory anyway with that disabled will throw the feature flag error.
Memory, no Pipeline: you can use this memory implementation for your crazy langchain app that lives in come other ecosystem entirely.

"Memory" is the most fundamental piece and it can be used independently by any application or a REST client. You CAN use RAG without memory as well if you are not passing a conversationId. I think we need to make sure we document these use cases and scenarios as part of the 2.10.0 release.

@HenryL27 could you please explain Trying to use memory anyway with that disabled will throw the feature flag error. --> I assume customer will not see this error?

@dhrubo-os this is the error we throw:

new OpenSearchException( "The experimental Conversation Memory feature is not enabled. To enable, please update the setting " + ConversationalIndexConstants.ML_COMMONS_MEMORY_FEATURE_ENABLED.getKey()

Sorry for the confusion. I'm trying to understand it step by step:

I enabled ML_COMMONS_RAG_PIPELINE_FEATURE_ENABLED

I asked a question in the RAG pipeline now. Can I continue my Q/A interaction with LLM now? Or it will show me this OpenSearchException to enable the other feature flag?

It won't. You will only get the error if you pass it a conversionId.

dhrubo-os · 2023-09-05T17:30:14Z

...g/opensearch/searchpipelines/questionanswering/generative/GenerativeQAResponseProcessor.java

@@ -115,6 +132,12 @@ private List<String> getSearchResults(SearchResponse response) {
        return searchResults;
    }

+    private static String jsonArrayToString(List<String> listOfStrings) {
+        JsonArray array = new JsonArray(listOfStrings.size());
+        listOfStrings.forEach(array::add);


do we need this line?

This is used to construct a single string out of an array as a helper method to store the search results as a "additionalInfo" string in memory.

...ensearch/searchpipelines/questionanswering/generative/client/ConversationalMemoryClient.java

dhrubo-os · 2023-09-05T17:38:30Z

...main/java/org/opensearch/searchpipelines/questionanswering/generative/prompt/PromptUtil.java

 * TODO Should prompt engineering llm-specific?
 *
 */
 @NoArgsConstructor(access = AccessLevel.PRIVATE)
 public class PromptUtil {

+    public static final String DEFAULT_CHAT_COMPLETION_PROMPT_TEMPLATE =


I remember you said you did some experiments on this prompt template to get the better result from LLM. I'm just curious to know more about that experiments. If you can share any doc/reference, that'll be helpful.

Ack. @HenryL27 can provide more details on this (but perhaps outside this PR?)

Yeah, sure. I think that's not going to block this PR.

austintlee · 2023-09-05T18:35:56Z

@dhrubo-os

| Exception in thread "main" java.lang.IllegalStateException: jar hell!
| class: org.bouncycastle.LICENSE
| jar1: /Users/taehe/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bcprov-jdk15to18/1.75/df22e1b6a9f6b218913f5b68dd16641344397fe0/bcprov-jdk15to18-1.75.jar
| jar2: /Users/taehe/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bcprov-ext-jdk15on/1.70/373d425c5ecb4edc9e3e2f7f7ff39bc8eff4abbf/bcprov-ext-jdk15on-1.70.jar
|       at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316)
|       at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215)
|       at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102)
|       at org.opensearch.bootstrap.JarHell.main(JarHell.java:86)

This seems unrelated to our change, though.

plugin/src/main/java/org/opensearch/ml/plugin/MachineLearningPlugin.java

...ensearch/searchpipelines/questionanswering/generative/client/ConversationalMemoryClient.java

jngz-es · 2023-09-05T18:19:49Z

...ensearch/searchpipelines/questionanswering/generative/client/ConversationalMemoryClient.java

+        return res.getId();
+    }
+
+    public List<Interaction> getInteractions(String conversationId, int lastN) {


I think we should have a generic getInteractions function here to allow some different algorithms to retrieve interactions, not only last N algorithm. But I am OK for now, we can refactor later to support new algorithm.

Yeah, let's do that for 2.11.

...ensearch/searchpipelines/questionanswering/generative/client/ConversationalMemoryClient.java

HenryL27 · 2023-09-05T19:10:14Z

@dhrubo-os

| Exception in thread "main" java.lang.IllegalStateException: jar hell!
| class: org.bouncycastle.LICENSE
| jar1: /Users/taehe/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bcprov-jdk15to18/1.75/df22e1b6a9f6b218913f5b68dd16641344397fe0/bcprov-jdk15to18-1.75.jar
| jar2: /Users/taehe/.gradle/caches/modules-2/files-2.1/org.bouncycastle/bcprov-ext-jdk15on/1.70/373d425c5ecb4edc9e3e2f7f7ff39bc8eff4abbf/bcprov-ext-jdk15on-1.70.jar
|       at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:316)
|       at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:215)
|       at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:102)
|       at org.opensearch.bootstrap.JarHell.main(JarHell.java:86)

This seems unrelated to our change, though.

This is also broken in 2.x

settings.gradle

Signed-off-by: Austin Lee <[email protected]>

dhrubo-os

Thanks for adding unit tests.

…earch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]>

…earch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>

* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]>

* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612)

* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Co-authored-by: HenryL27 <[email protected]>

* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>

* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation backport to 2.x (#1286) * Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]> * fix http library version Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]>

austintlee added 2 commits September 1, 2023 13:34

Put RAG pipeline behind a feature flag.

1baa080

Signed-off-by: Austin Lee <[email protected]>

Add support for chat history in RAG using the Conversational Memory API

1eb9444

Signed-off-by: Austin Lee <[email protected]>

austintlee had a problem deploying to ml-commons-cicd-env September 1, 2023 21:02 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 1, 2023 21:02 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 1, 2023 21:02 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 1, 2023 21:02 — with GitHub Actions Failure

Fix spotless

8f4d90d

Signed-off-by: Austin Lee <[email protected]>

austintlee temporarily deployed to ml-commons-cicd-env September 1, 2023 21:15 — with GitHub Actions Inactive

austintlee had a problem deploying to ml-commons-cicd-env September 1, 2023 21:15 — with GitHub Actions Failure

austintlee temporarily deployed to ml-commons-cicd-env September 1, 2023 21:15 — with GitHub Actions Inactive

Fix RAG feature flag enablement.

a1df6a1

Signed-off-by: Austin Lee <[email protected]>

austintlee had a problem deploying to ml-commons-cicd-env September 4, 2023 15:54 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 4, 2023 15:54 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 4, 2023 15:54 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 4, 2023 15:54 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 17:41 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 17:41 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 17:41 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 17:41 — with GitHub Actions Failure

dhrubo-os reviewed Sep 5, 2023

View reviewed changes

jngz-es reviewed Sep 5, 2023

View reviewed changes

austintlee commented Sep 5, 2023

View reviewed changes

settings.gradle Show resolved Hide resolved

Address comments.

8cbac4c

Signed-off-by: Austin Lee <[email protected]>

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 21:22 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 21:22 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 21:22 — with GitHub Actions Error

Add unit tests for MachineLearningPlugin

739d96a

Signed-off-by: Austin Lee <[email protected]>

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:34 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:34 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:34 — with GitHub Actions Error

dhrubo-os approved these changes Sep 5, 2023

View reviewed changes

jngz-es approved these changes Sep 5, 2023

View reviewed changes

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:50 — with GitHub Actions Error

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:50 — with GitHub Actions Failure

austintlee had a problem deploying to ml-commons-cicd-env September 5, 2023 22:50 — with GitHub Actions Error

dhrubo-os merged commit 180c791 into opensearch-project:feature/conversation Sep 5, 2023

This was referenced Sep 5, 2023

Move ConversationalMemoryClient to ml-client #1282

Open

[FEATURE] ConversationalMemoryClient - different ways to retrieve relevant interactions #1284

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Add Retrieval Augmented Generation search processors #1275

[Feature] Add Retrieval Augmented Generation search processors #1275

austintlee commented Sep 1, 2023

austintlee commented Sep 1, 2023

austintlee commented Sep 1, 2023

codecov bot commented Sep 1, 2023 •

edited

Loading

austintlee commented Sep 4, 2023

dhrubo-os commented Sep 5, 2023

dhrubo-os Sep 5, 2023

HenryL27 Sep 5, 2023

austintlee Sep 5, 2023

dhrubo-os Sep 5, 2023

austintlee Sep 5, 2023

dhrubo-os Sep 5, 2023

austintlee Sep 5, 2023

dhrubo-os Sep 5, 2023

austintlee Sep 5, 2023

dhrubo-os Sep 5, 2023

austintlee Sep 5, 2023

dhrubo-os Sep 5, 2023 •

edited

Loading

austintlee commented Sep 5, 2023

jngz-es Sep 5, 2023

HenryL27 Sep 5, 2023

austintlee Sep 5, 2023

HenryL27 commented Sep 5, 2023

dhrubo-os left a comment

[Feature] Add Retrieval Augmented Generation search processors #1275

[Feature] Add Retrieval Augmented Generation search processors #1275

Conversation

austintlee commented Sep 1, 2023

Description

Issues Resolved

Check List

austintlee commented Sep 1, 2023

austintlee commented Sep 1, 2023

codecov bot commented Sep 1, 2023 • edited Loading

Codecov Report

austintlee commented Sep 4, 2023

dhrubo-os commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dhrubo-os Sep 5, 2023 • edited Loading

Choose a reason for hiding this comment

austintlee commented Sep 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HenryL27 commented Sep 5, 2023

dhrubo-os left a comment

Choose a reason for hiding this comment

codecov bot commented Sep 1, 2023 •

edited

Loading

dhrubo-os Sep 5, 2023 •

edited

Loading