Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase #626

tileintel · 2024-09-05T18:15:14Z

Description

Add support/gateway for MMRAG MultimodalRAGQnAWithVideos usecase.
The new MultimodalRAGQnAWithVideosGateway is initialized with 2 ServiceOrchestrators. One is to handle MMRAG for the query that requires multimodal retrieval results. The second one is simply LVM ServiceOrchestrator to handle follow-up queries that do not require multimodal retrieval results.

Issues

RFC: https://github.com/opea-project/docs/pull/49/files
Issue: opea-project/GenAIExamples#358

Type of change

List the type of change like below. Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
[x ] New feature (non-breaking change which adds new functionality)
Breaking change (fix or feature that would break existing design and interface)
Others (enhancement, documentation, validation, etc.)

Dependencies

None

Tests

We have provided 7 unittests in tests/cores/mega/test_multimodalrag_qna_with_videos_gateway.py: 2 of them test the two ServiceOrchestrator used in the new gateway, and the other 5 test the new gateway for different different scenarios

Signed-off-by: Tiep Le <[email protected]>

Signed-off-by: siddhivelankar23 <[email protected]>

merge with main

Signed-off-by: siddhivelankar23 <[email protected]>

Signed-off-by: sjagtap1803 <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: sjagtap1803 <[email protected]>

Retriever lvm update

Signed-off-by: sjagtap1803 <[email protected]>

for more information, see https://pre-commit.ci

Retriever lvm update

Signed-off-by: siddhivelankar23 <[email protected]>

Signed-off-by: sjagtap1803 <[email protected]>

Signed-off-by: siddhivelankar23 <[email protected]>

for more information, see https://pre-commit.ci

Retriever lvm update

codecov · 2024-09-05T18:29:17Z

Codecov Report

Attention: Patch coverage is 76.59574% with 22 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
comps/cores/mega/gateway.py	76.34%	22 Missing ⚠️

Files with missing lines	Coverage Δ
comps/cores/mega/constants.py	`98.21% <100.00%> (+0.03%)`	⬆️
comps/cores/mega/gateway.py	`35.90% <76.34%> (+13.28%)`	⬆️

... and 1 file with indirect coverage changes

Signed-off-by: Tiep Le <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Tiep Le <[email protected]>

…eintel/GenAIComps into mmrag_videos_megaservice-dev

Signed-off-by: Tiep Le <[email protected]>

for more information, see https://pre-commit.ci

Signed-off-by: Tiep Le <[email protected]>

…eintel/GenAIComps into mmrag_videos_megaservice-dev

XuhuiRen · 2024-09-06T01:56:50Z

what is the difference with #606?

tileintel · 2024-09-06T05:25:43Z

what is the difference with #606?

This PR is developed starting from #606. So can you please click merge #606 first, and then review this PR? Then this PR will be much less file changed. Sorry for the confusion. There is no change from #606 in this PR.

opea-project#626) * updates Signed-off-by: Tiep Le <[email protected]> * cosmetic Signed-off-by: siddhivelankar23 <[email protected]> * update redis schema Signed-off-by: siddhivelankar23 <[email protected]> * update multimodal config and docker compose retriever Signed-off-by: siddhivelankar23 <[email protected]> * update requirements Signed-off-by: siddhivelankar23 <[email protected]> * update retriever redis Signed-off-by: siddhivelankar23 <[email protected]> * multimodal retriever implementation Signed-off-by: siddhivelankar23 <[email protected]> * test for multimodal retriever Signed-off-by: siddhivelankar23 <[email protected]> * include prompt preparation for multimodal rag on videos application Signed-off-by: sjagtap1803 <[email protected]> * fix template Signed-off-by: sjagtap1803 <[email protected]> * add test for llava for mm_rag_on_videos Signed-off-by: sjagtap1803 <[email protected]> * update test Signed-off-by: sjagtap1803 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * first update on gateaway Signed-off-by: sjagtap1803 <[email protected]> * fix index not found Signed-off-by: sjagtap1803 <[email protected]> * add LVMSearchedMultimodalDoc Signed-off-by: sjagtap1803 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement gateway for MultimodalRagQnAWithVideos Signed-off-by: siddhivelankar23 <[email protected]> * remove INDEX_SCHEMA Signed-off-by: siddhivelankar23 <[email protected]> * update MultimodalRAGQnAWithVideosGateway with 2 megaservices Signed-off-by: sjagtap1803 <[email protected]> * revise folder structure to comps/retrievers/langchain/redis_multimodal Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * add unittest for multimodalrag_qna_with_videos_gateway Signed-off-by: siddhivelankar23 <[email protected]> * update test mmrag qna with videos Signed-off-by: Tiep Le <[email protected]> * change port of redis to resolve CI test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update lvms test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update test for multimodal rag qna with videos gateway Signed-off-by: siddhivelankar23 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add more test to increase coverage Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cosmetic Signed-off-by: Tiep Le <[email protected]> * add more test Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update name of gateway Signed-off-by: Tiep Le <[email protected]> --------- Signed-off-by: Tiep Le <[email protected]> Signed-off-by: siddhivelankar23 <[email protected]> Signed-off-by: sjagtap1803 <[email protected]> Co-authored-by: siddhivelankar23 <[email protected]> Co-authored-by: sjagtap1803 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]>

…opea-project#626) * updates Signed-off-by: Tiep Le <[email protected]> * cosmetic Signed-off-by: siddhivelankar23 <[email protected]> * update redis schema Signed-off-by: siddhivelankar23 <[email protected]> * update multimodal config and docker compose retriever Signed-off-by: siddhivelankar23 <[email protected]> * update requirements Signed-off-by: siddhivelankar23 <[email protected]> * update retriever redis Signed-off-by: siddhivelankar23 <[email protected]> * multimodal retriever implementation Signed-off-by: siddhivelankar23 <[email protected]> * test for multimodal retriever Signed-off-by: siddhivelankar23 <[email protected]> * include prompt preparation for multimodal rag on videos application Signed-off-by: sjagtap1803 <[email protected]> * fix template Signed-off-by: sjagtap1803 <[email protected]> * add test for llava for mm_rag_on_videos Signed-off-by: sjagtap1803 <[email protected]> * update test Signed-off-by: sjagtap1803 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * first update on gateaway Signed-off-by: sjagtap1803 <[email protected]> * fix index not found Signed-off-by: sjagtap1803 <[email protected]> * add LVMSearchedMultimodalDoc Signed-off-by: sjagtap1803 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement gateway for MultimodalRagQnAWithVideos Signed-off-by: siddhivelankar23 <[email protected]> * remove INDEX_SCHEMA Signed-off-by: siddhivelankar23 <[email protected]> * update MultimodalRAGQnAWithVideosGateway with 2 megaservices Signed-off-by: sjagtap1803 <[email protected]> * revise folder structure to comps/retrievers/langchain/redis_multimodal Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * add unittest for multimodalrag_qna_with_videos_gateway Signed-off-by: siddhivelankar23 <[email protected]> * update test mmrag qna with videos Signed-off-by: Tiep Le <[email protected]> * change port of redis to resolve CI test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update lvms test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update test Signed-off-by: siddhivelankar23 <[email protected]> * update test for multimodal rag qna with videos gateway Signed-off-by: siddhivelankar23 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add more test to increase coverage Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cosmetic Signed-off-by: Tiep Le <[email protected]> * add more test Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update name of gateway Signed-off-by: Tiep Le <[email protected]> --------- Signed-off-by: Tiep Le <[email protected]> Signed-off-by: siddhivelankar23 <[email protected]> Signed-off-by: sjagtap1803 <[email protected]> Co-authored-by: siddhivelankar23 <[email protected]> Co-authored-by: sjagtap1803 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]> Signed-off-by: sharanshirodkar7 <[email protected]>

Signed-off-by: Yingchun Guo <[email protected]>

tileintel and others added 30 commits August 29, 2024 22:00

updates

ac30d1a

Signed-off-by: Tiep Le <[email protected]>

cosmetic

77632f2

Signed-off-by: siddhivelankar23 <[email protected]>

Merge pull request #4 from tileintel/main

049c78d

merge with main

update redis schema

4c3ea33

Signed-off-by: siddhivelankar23 <[email protected]>

update multimodal config and docker compose retriever

1688033

Signed-off-by: siddhivelankar23 <[email protected]>

update requirements

bc1699f

Signed-off-by: siddhivelankar23 <[email protected]>

update retriever redis

5ecbfdb

Signed-off-by: siddhivelankar23 <[email protected]>

multimodal retriever implementation

bc23290

Signed-off-by: siddhivelankar23 <[email protected]>

test for multimodal retriever

c56151d

Signed-off-by: siddhivelankar23 <[email protected]>

include prompt preparation for multimodal rag on videos application

4c26828

Signed-off-by: sjagtap1803 <[email protected]>

fix template

fefcf72

Signed-off-by: sjagtap1803 <[email protected]>

add test for llava for mm_rag_on_videos

4280393

Signed-off-by: sjagtap1803 <[email protected]>

update test

3c4b505

Signed-off-by: sjagtap1803 <[email protected]>

Merge branch 'main' into retriever_lvm_update

70993d9

[pre-commit.ci] auto fixes from pre-commit.com hooks

5aa4c02

for more information, see https://pre-commit.ci

first update on gateaway

542e6cd

Signed-off-by: sjagtap1803 <[email protected]>

fix index not found

01b2138

Signed-off-by: sjagtap1803 <[email protected]>

Merge branch 'main' into retriever_lvm_update

08ef0ec

Merge branch 'main' into retriever_lvm_update

3c6e471

Merge pull request #5 from tileintel/retriever_lvm_update

1d5c67b

Retriever lvm update

add LVMSearchedMultimodalDoc

4d353e8

Signed-off-by: sjagtap1803 <[email protected]>

Merge branch 'main' into retriever_lvm_update

b69cef1

[pre-commit.ci] auto fixes from pre-commit.com hooks

ac57ca0

for more information, see https://pre-commit.ci

Merge pull request #6 from tileintel/retriever_lvm_update

78d3902

Retriever lvm update

implement gateway for MultimodalRagQnAWithVideos

2c1f0ba

Signed-off-by: siddhivelankar23 <[email protected]>

remove INDEX_SCHEMA

dc631db

Signed-off-by: siddhivelankar23 <[email protected]>

Merge branch 'main' into retriever_lvm_update

ccf3da7

Merge branch 'main' into retriever_lvm_update

0965882

update MultimodalRAGQnAWithVideosGateway with 2 megaservices

5a3c764

Signed-off-by: sjagtap1803 <[email protected]>

revise folder structure to comps/retrievers/langchain/redis_multimodal

5842ed1

Signed-off-by: siddhivelankar23 <[email protected]>

siddhivelankar23 added 3 commits September 5, 2024 11:06

update test

1d8598e

Signed-off-by: siddhivelankar23 <[email protected]>

update test

b757b39

Signed-off-by: siddhivelankar23 <[email protected]>

update test for multimodal rag qna with videos gateway

98afd97

Signed-off-by: siddhivelankar23 <[email protected]>

tileintel requested review from XuhuiRen, Spycsh and lvliang-intel as code owners September 5, 2024 18:15

tileintel and others added 3 commits September 5, 2024 11:15

Merge branch 'main' into mmrag_videos_megaservice-dev

d15d430

[pre-commit.ci] auto fixes from pre-commit.com hooks

407ff2b

for more information, see https://pre-commit.ci

Merge pull request #8 from tileintel/retriever_lvm_update

fb0b834

Retriever lvm update

tileintel and others added 10 commits September 5, 2024 18:40

add more test to increase coverage

7478b06

Signed-off-by: Tiep Le <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

117b8f2

for more information, see https://pre-commit.ci

cosmetic

b787eda

Signed-off-by: Tiep Le <[email protected]>

Merge branch 'mmrag_videos_megaservice-dev' of https://github.com/til…

be9124f

…eintel/GenAIComps into mmrag_videos_megaservice-dev

add more test

f9fd4b4

Signed-off-by: Tiep Le <[email protected]>

Merge branch 'main' into mmrag_videos_megaservice-dev

5e899c7

[pre-commit.ci] auto fixes from pre-commit.com hooks

50dcd31

for more information, see https://pre-commit.ci

update name of gateway

693b1c9

Signed-off-by: Tiep Le <[email protected]>

Merge branch 'mmrag_videos_megaservice-dev' of https://github.com/til…

9b79373

…eintel/GenAIComps into mmrag_videos_megaservice-dev

Merge branch 'main' into mmrag_videos_megaservice-dev

c8ee3e8

Spycsh and others added 2 commits September 6, 2024 13:46

Merge branch 'main' into mmrag_videos_megaservice-dev

241cd31

Merge branch 'main' into mmrag_videos_megaservice-dev

76109ef

lvliang-intel approved these changes Sep 6, 2024

View reviewed changes

kevinintel approved these changes Sep 6, 2024

View reviewed changes

kevinintel merged commit 99be1bd into opea-project:main Sep 6, 2024
11 checks passed

lkk12014402 pushed a commit that referenced this pull request Sep 19, 2024

update benchmark manifest to fix errors (#626)

4fd3517

Signed-off-by: Yingchun Guo <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase #626

Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase #626

tileintel commented Sep 5, 2024 •

edited

Loading

codecov bot commented Sep 5, 2024 •

edited

Loading

XuhuiRen commented Sep 6, 2024

tileintel commented Sep 6, 2024 •

edited

Loading

Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase #626

Add Megaservice support for MMRAG - MultimodalRAGQnAWithVideos usecase #626

Conversation

tileintel commented Sep 5, 2024 • edited Loading

Description

Issues

Type of change

Dependencies

Tests

codecov bot commented Sep 5, 2024 • edited Loading

Codecov Report

XuhuiRen commented Sep 6, 2024

tileintel commented Sep 6, 2024 • edited Loading

tileintel commented Sep 5, 2024 •

edited

Loading

codecov bot commented Sep 5, 2024 •

edited

Loading

tileintel commented Sep 6, 2024 •

edited

Loading