-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/conversation memory feature flag #1271
Feature/conversation memory feature flag #1271
Conversation
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
@HenryL27 can you have a look at the failing test? |
@dhrubo-os @ylwu-amzn can you have a look at the code change here? Thanks! |
the failing test is in |
Signed-off-by: HenryL27 <[email protected]>
.../main/java/org/opensearch/ml/memory/action/conversation/GetConversationsTransportAction.java
Outdated
Show resolved
Hide resolved
Codecov Report
@@ Coverage Diff @@
## feature/conversation #1271 +/- ##
==========================================================
- Coverage 79.04% 78.99% -0.06%
+ Complexity 2148 2146 -2
==========================================================
Files 172 172
Lines 8672 8673 +1
Branches 870 870
==========================================================
- Hits 6855 6851 -4
- Misses 1422 1428 +6
+ Partials 395 394 -1
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…ning Signed-off-by: HenryL27 <[email protected]>
include 'conversational-memory' | ||
project(":conversational-memory").name = rootProject.name + "-memory" | ||
include 'memory' | ||
project(":memory").name = rootProject.name + "-memory" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dhrubo-os ^^
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]>
* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612)
* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Co-authored-by: HenryL27 <[email protected]>
* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>
* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]>
* Feature/conversation backport to 2.x (#1286) * Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]> * fix http library version Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]>
Description
Block memory transport actions behind feature flag
Issues Resolved
feature flag is needed for experimental features
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.