Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/conversation backport to 2.x #1286

Merged

Conversation

HenryL27
Copy link
Collaborator

@HenryL27 HenryL27 commented Sep 5, 2023

Description

Bring RAG pipeline and Conversation memory API into 2.x

Issues Resolved

#1150

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@HenryL27
Copy link
Collaborator Author

HenryL27 commented Sep 6, 2023

#1288 will fix jarhell (should I just add those changes here independently too?)

@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 03:28 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 03:28 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 03:28 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 03:28 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Sep 7, 2023

Codecov Report

Merging #1286 (b670854) into 2.x (86eb953) will increase coverage by 0.49%.
The diff coverage is 94.01%.

@@             Coverage Diff              @@
##                2.x    #1286      +/-   ##
============================================
+ Coverage     78.86%   79.36%   +0.49%     
- Complexity     2142     2246     +104     
============================================
  Files           168      188      +20     
  Lines          8741     9074     +333     
  Branches        877      903      +26     
============================================
+ Hits           6894     7202     +308     
- Misses         1446     1464      +18     
- Partials        401      408       +7     
Flag Coverage Δ
ml-commons 79.36% <94.01%> (+0.49%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed Coverage Δ
...nerative/client/MachineLearningInternalClient.java 72.41% <72.41%> (ø)
...ing/generative/GenerativeQAProcessorConstants.java 75.00% <75.00%> (ø)
...rg/opensearch/ml/plugin/MachineLearningPlugin.java 96.18% <84.61%> (-2.88%) ⬇️
...nswering/generative/ext/GenerativeQAParamUtil.java 88.88% <88.88%> (ø)
...ring/generative/GenerativeQAResponseProcessor.java 91.07% <91.07%> (ø)
.../generative/client/ConversationalMemoryClient.java 96.87% <96.87%> (ø)
...ch/ml/rest/RestMemoryCreateConversationAction.java 100.00% <100.00%> (ø)
...rch/ml/rest/RestMemoryCreateInteractionAction.java 100.00% <100.00%> (ø)
...ch/ml/rest/RestMemoryDeleteConversationAction.java 100.00% <100.00%> (ø)
...arch/ml/rest/RestMemoryGetConversationsAction.java 100.00% <100.00%> (ø)
... and 12 more

... and 2 files with indirect coverage changes

📢 Have feedback on the report? Share it here.

HenryL27 and others added 6 commits September 7, 2023 11:54
* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per opensearch-project#1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
… enable Retrieval Augmented Generation (RAG) (opensearch-project#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (opensearch-project#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
…earch-project#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
opensearch-project#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
@HenryL27 HenryL27 force-pushed the feature/conversation-upstream branch from 943fc37 to b670854 Compare September 7, 2023 18:58
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 18:58 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 18:58 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 18:58 — with GitHub Actions Inactive
@HenryL27 HenryL27 temporarily deployed to ml-commons-cicd-env September 7, 2023 18:58 — with GitHub Actions Inactive
@dhrubo-os dhrubo-os merged commit 1112612 into opensearch-project:2.x Sep 7, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 7, 2023
* Conversational Memory for GenAI Apps (#1196)

* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per #1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation memory feature flag (#1271)

* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* [Feature] Add Retrieval Augmented Generation search processors (#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
(cherry picked from commit 1112612)
dhrubo-os pushed a commit that referenced this pull request Sep 7, 2023
* Conversational Memory for GenAI Apps (#1196)

* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per #1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation memory feature flag (#1271)

* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* [Feature] Add Retrieval Augmented Generation search processors (#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
(cherry picked from commit 1112612)

Co-authored-by: HenryL27 <[email protected]>
HenryL27 added a commit to HenryL27/ml-commons that referenced this pull request Oct 3, 2023
* Conversational Memory for GenAI Apps (opensearch-project#1196)

* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per opensearch-project#1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation memory feature flag (opensearch-project#1271)

* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (opensearch-project#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>
HenryL27 added a commit to HenryL27/ml-commons that referenced this pull request Oct 3, 2023
* Conversational Memory for GenAI Apps (opensearch-project#1196)

* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per opensearch-project#1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation memory feature flag (opensearch-project#1271)

* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (opensearch-project#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (opensearch-project#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* [Feature] Add Retrieval Augmented Generation search processors (opensearch-project#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* Allow RAG pipeline feature flag to be enabled and disabled dynamically (opensearch-project#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
(cherry picked from commit 1112612)
Signed-off-by: HenryL27 <[email protected]>
dhrubo-os pushed a commit that referenced this pull request Oct 4, 2023
* Feature/conversation backport to 2.x (#1286)

* Conversational Memory for GenAI Apps (#1196)

* moved code over

Signed-off-by: HenryL27 <[email protected]>

* added actions to MLPlugin; fixed io lib stuff

Signed-off-by: HenryL27 <[email protected]>

* fixed copyrights again

Signed-off-by: HenryL27 <[email protected]>

* Fix nullptr exception in .equals

Signed-off-by: HenryL27 <[email protected]>

* preserve thread context across action calls

Signed-off-by: HenryL27 <[email protected]>

* remove MissingResourceException from CreatInteractionRequest in favor of IOException

Signed-off-by: HenryL27 <[email protected]>

* move ConversationMet, Interaction, and Constants to common/conversational

Signed-off-by: HenryL27 <[email protected]>

* Sequentialize createInteraction to remove data race

Signed-off-by: HenryL27 <[email protected]>

* allow disorder when conversations have same timestamp

Signed-off-by: HenryL27 <[email protected]>

* lombokify

Signed-off-by: HenryL27 <[email protected]>

* add some unit testing

Signed-off-by: HenryL27 <[email protected]>

* Increase unit test coverage

Signed-off-by: HenryL27 <[email protected]>

* fix naming

Signed-off-by: HenryL27 <[email protected]>

* finish code coverage for actions

Signed-off-by: HenryL27 <[email protected]>

* Leave null values out of XContent per #1196 (comment)

Signed-off-by: HenryL27 <[email protected]>

* Add integration tests for rest actions

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* Complete unit testing for Index classes

Signed-off-by: HenryL27 <[email protected]>

* update build.gradle

Signed-off-by: HenryL27 <[email protected]>

* Finish unit tests

Signed-off-by: HenryL27 <[email protected]>

* Fail closed on missing convo access

Signed-off-by: HenryL27 <[email protected]>

* address code review/walkthrough comments

Signed-off-by: HenryL27 <[email protected]>

* re-add prompt temlplate and metadata fields at interaction level

Signed-off-by: HenryL27 <[email protected]>

* parse request body, not params, for post requests

Signed-off-by: HenryL27 <[email protected]>

* restructure with memory as higher-level term

Signed-off-by: HenryL27 <[email protected]>

* clean up build.gradle

Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

* change interaction field names
timestamp -> create_time
metadata -> additional_info

Signed-off-by: HenryL27 <[email protected]>

* fix GetInteractionsResponse xcontent tests

Signed-off-by: HenryL27 <[email protected]>

* propagate name change to variables and parameters

Signed-off-by: HenryL27 <[email protected]>

* clean logging and fix typos

Signed-off-by: HenryL27 <[email protected]>

* fix final convtructor according to find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* append plugin-ml- to index names

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Feature/conversation memory feature flag (#1271)

* add feature flag and checks to transport actions

Signed-off-by: HenryL27 <[email protected]>

* add feature flag tests

Signed-off-by: HenryL27 <[email protected]>

* fix typos for real with find-and-replace

Signed-off-by: HenryL27 <[email protected]>

* rename conversational-memory directory to memory

Signed-off-by: HenryL27 <[email protected]>

* fix settings.gradle with new dir name

Signed-off-by: HenryL27 <[email protected]>

* re-add feature flag checks and tests to transport layer

Signed-off-by: HenryL27 <[email protected]>

* fix feature flag with updateConsumer

Signed-off-by: HenryL27 <[email protected]>

* remove redundant settings update

Signed-off-by: HenryL27 <[email protected]>

* clean up feature var initialization to avoid unchecked conversion warning

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>

* Use Search Pipeline processors, Remote Inference and HttpConnector to enable Retrieval Augmented Generation (RAG) (#1195)

* Use Search Pipeline processors, Remote Inference and HttpConnector to
enable Retrieval Augmented Generation (RAG) (#1150)

Signed-off-by: Austin Lee <[email protected]>

* Address test coverage.

Signed-off-by: Austin Lee <[email protected]>

* Fix/update imports due to changes coming from core.

Signed-off-by: Austin Lee <[email protected]>

* Update license header.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Use List for context fields so we can pull contexts from multiple fields when constructing contexts for LLMs.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless issue.

Signed-off-by: Austin Lee <[email protected]>

* Update README.

Signed-off-by: Austin Lee <[email protected]>

* Fix ml-client shadowJar implicit dependency issue.

Signed-off-by: Austin Lee <[email protected]>

* Add a wrapper client for ML predict.

Signed-off-by: Austin Lee <[email protected]>

* Add tests for the internal ML client.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* [Feature] Add Retrieval Augmented Generation search processors (#1275)

* Put RAG pipeline behind a feature flag.

Signed-off-by: Austin Lee <[email protected]>

* Add support for chat history in RAG using the Conversational Memory API

Signed-off-by: Austin Lee <[email protected]>

* Fix spotless

Signed-off-by: Austin Lee <[email protected]>

* Fix RAG feature flag enablement.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments and suggestions.

Signed-off-by: Austin Lee <[email protected]>

* Address comments.

Signed-off-by: Austin Lee <[email protected]>

* Add unit tests for MachineLearningPlugin

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* Allow RAG pipeline feature flag to be enabled and disabled dynamically (#1293)

* Allow RAG pipeline feature flag to be enabled and disabled dynamically.

Signed-off-by: Austin Lee <[email protected]>

* Address review comments.

Signed-off-by: Austin Lee <[email protected]>

* Add negative test cases for RAG feature flag being turned off.

Signed-off-by: Austin Lee <[email protected]>

* Improve error checking.

Signed-off-by: Austin Lee <[email protected]>

---------

Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: HenryL27 <[email protected]>

* apply spotless

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
(cherry picked from commit 1112612)
Signed-off-by: HenryL27 <[email protected]>

* fix http library version

Signed-off-by: HenryL27 <[email protected]>

---------

Signed-off-by: HenryL27 <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Signed-off-by: Austin Lee <[email protected]>
Co-authored-by: Austin Lee <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants