-
Notifications
You must be signed in to change notification settings - Fork 138
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/conversation backport to 2.x #1286
Merged
dhrubo-os
merged 6 commits into
opensearch-project:2.x
from
HenryL27:feature/conversation-upstream
Sep 7, 2023
Merged
Feature/conversation backport to 2.x #1286
dhrubo-os
merged 6 commits into
opensearch-project:2.x
from
HenryL27:feature/conversation-upstream
Sep 7, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 5, 2023 23:58 — with
GitHub Actions
Failure
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 5, 2023 23:58 — with
GitHub Actions
Error
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 5, 2023 23:58 — with
GitHub Actions
Failure
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 5, 2023 23:58 — with
GitHub Actions
Error
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 6, 2023 23:01 — with
GitHub Actions
Error
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 6, 2023 23:01 — with
GitHub Actions
Failure
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 6, 2023 23:01 — with
GitHub Actions
Error
HenryL27
had a problem deploying
to
ml-commons-cicd-env
September 6, 2023 23:01 — with
GitHub Actions
Failure
#1288 will fix jarhell (should I just add those changes here independently too?) |
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 03:28 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 03:28 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 03:28 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 03:28 — with
GitHub Actions
Inactive
Codecov Report
@@ Coverage Diff @@
## 2.x #1286 +/- ##
============================================
+ Coverage 78.86% 79.36% +0.49%
- Complexity 2142 2246 +104
============================================
Files 168 188 +20
Lines 8741 9074 +333
Branches 877 903 +26
============================================
+ Hits 6894 7202 +308
- Misses 1446 1464 +18
- Partials 401 408 +7
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 📢 Have feedback on the report? Share it here. |
* moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]>
… enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>
…earch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>
opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
HenryL27
force-pushed
the
feature/conversation-upstream
branch
from
September 7, 2023 18:58
943fc37
to
b670854
Compare
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 18:58 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 18:58 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 18:58 — with
GitHub Actions
Inactive
HenryL27
temporarily deployed
to
ml-commons-cicd-env
September 7, 2023 18:58 — with
GitHub Actions
Inactive
ylwu-amzn
approved these changes
Sep 7, 2023
dhrubo-os
approved these changes
Sep 7, 2023
opensearch-trigger-bot bot
pushed a commit
that referenced
this pull request
Sep 7, 2023
* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612)
dhrubo-os
pushed a commit
that referenced
this pull request
Sep 7, 2023
* Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Co-authored-by: HenryL27 <[email protected]>
HenryL27
added a commit
to HenryL27/ml-commons
that referenced
this pull request
Oct 3, 2023
* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]>
HenryL27
added a commit
to HenryL27/ml-commons
that referenced
this pull request
Oct 3, 2023
* Conversational Memory for GenAI Apps (opensearch-project#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per opensearch-project#1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (opensearch-project#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]>
5 tasks
dhrubo-os
pushed a commit
that referenced
this pull request
Oct 4, 2023
* Feature/conversation backport to 2.x (#1286) * Conversational Memory for GenAI Apps (#1196) * moved code over Signed-off-by: HenryL27 <[email protected]> * added actions to MLPlugin; fixed io lib stuff Signed-off-by: HenryL27 <[email protected]> * fixed copyrights again Signed-off-by: HenryL27 <[email protected]> * Fix nullptr exception in .equals Signed-off-by: HenryL27 <[email protected]> * preserve thread context across action calls Signed-off-by: HenryL27 <[email protected]> * remove MissingResourceException from CreatInteractionRequest in favor of IOException Signed-off-by: HenryL27 <[email protected]> * move ConversationMet, Interaction, and Constants to common/conversational Signed-off-by: HenryL27 <[email protected]> * Sequentialize createInteraction to remove data race Signed-off-by: HenryL27 <[email protected]> * allow disorder when conversations have same timestamp Signed-off-by: HenryL27 <[email protected]> * lombokify Signed-off-by: HenryL27 <[email protected]> * add some unit testing Signed-off-by: HenryL27 <[email protected]> * Increase unit test coverage Signed-off-by: HenryL27 <[email protected]> * fix naming Signed-off-by: HenryL27 <[email protected]> * finish code coverage for actions Signed-off-by: HenryL27 <[email protected]> * Leave null values out of XContent per #1196 (comment) Signed-off-by: HenryL27 <[email protected]> * Add integration tests for rest actions Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * Complete unit testing for Index classes Signed-off-by: HenryL27 <[email protected]> * update build.gradle Signed-off-by: HenryL27 <[email protected]> * Finish unit tests Signed-off-by: HenryL27 <[email protected]> * Fail closed on missing convo access Signed-off-by: HenryL27 <[email protected]> * address code review/walkthrough comments Signed-off-by: HenryL27 <[email protected]> * re-add prompt temlplate and metadata fields at interaction level Signed-off-by: HenryL27 <[email protected]> * parse request body, not params, for post requests Signed-off-by: HenryL27 <[email protected]> * restructure with memory as higher-level term Signed-off-by: HenryL27 <[email protected]> * clean up build.gradle Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> * change interaction field names timestamp -> create_time metadata -> additional_info Signed-off-by: HenryL27 <[email protected]> * fix GetInteractionsResponse xcontent tests Signed-off-by: HenryL27 <[email protected]> * propagate name change to variables and parameters Signed-off-by: HenryL27 <[email protected]> * clean logging and fix typos Signed-off-by: HenryL27 <[email protected]> * fix final convtructor according to find-and-replace Signed-off-by: HenryL27 <[email protected]> * append plugin-ml- to index names Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Feature/conversation memory feature flag (#1271) * add feature flag and checks to transport actions Signed-off-by: HenryL27 <[email protected]> * add feature flag tests Signed-off-by: HenryL27 <[email protected]> * fix typos for real with find-and-replace Signed-off-by: HenryL27 <[email protected]> * rename conversational-memory directory to memory Signed-off-by: HenryL27 <[email protected]> * fix settings.gradle with new dir name Signed-off-by: HenryL27 <[email protected]> * re-add feature flag checks and tests to transport layer Signed-off-by: HenryL27 <[email protected]> * fix feature flag with updateConsumer Signed-off-by: HenryL27 <[email protected]> * remove redundant settings update Signed-off-by: HenryL27 <[email protected]> * clean up feature var initialization to avoid unchecked conversion warning Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195) * Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1150) Signed-off-by: Austin Lee <[email protected]> * Address test coverage. Signed-off-by: Austin Lee <[email protected]> * Fix/update imports due to changes coming from core. Signed-off-by: Austin Lee <[email protected]> * Update license header. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Fix spotless issue. Signed-off-by: Austin Lee <[email protected]> * Update README. Signed-off-by: Austin Lee <[email protected]> * Fix ml-client shadowJar implicit dependency issue. Signed-off-by: Austin Lee <[email protected]> * Add a wrapper client for ML predict. Signed-off-by: Austin Lee <[email protected]> * Add tests for the internal ML client. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * [Feature] Add Retrieval Augmented Generation search processors (#1275) * Put RAG pipeline behind a feature flag. Signed-off-by: Austin Lee <[email protected]> * Add support for chat history in RAG using the Conversational Memory API Signed-off-by: Austin Lee <[email protected]> * Fix spotless Signed-off-by: Austin Lee <[email protected]> * Fix RAG feature flag enablement. Signed-off-by: Austin Lee <[email protected]> * Address review comments and suggestions. Signed-off-by: Austin Lee <[email protected]> * Address comments. Signed-off-by: Austin Lee <[email protected]> * Add unit tests for MachineLearningPlugin Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293) * Allow RAG pipeline feature flag to be enabled and disabled dynamically. Signed-off-by: Austin Lee <[email protected]> * Address review comments. Signed-off-by: Austin Lee <[email protected]> * Add negative test cases for RAG feature flag being turned off. Signed-off-by: Austin Lee <[email protected]> * Improve error checking. Signed-off-by: Austin Lee <[email protected]> --------- Signed-off-by: Austin Lee <[email protected]> Signed-off-by: HenryL27 <[email protected]> * apply spotless Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]> (cherry picked from commit 1112612) Signed-off-by: HenryL27 <[email protected]> * fix http library version Signed-off-by: HenryL27 <[email protected]> --------- Signed-off-by: HenryL27 <[email protected]> Signed-off-by: Austin Lee <[email protected]> Signed-off-by: Austin Lee <[email protected]> Co-authored-by: Austin Lee <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Bring RAG pipeline and Conversation memory API into 2.x
Issues Resolved
#1150
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.