adding embedding support for CLIP based models for VideoRAGQnA #533

srinarayan-srikanthan · 2024-08-20T22:11:31Z

Description

Add support for CLIP based embeddings for multimodal VideoRAG usecase.

Issues

List the issue or RFC link this PR is working on. If there is no such link, please mark it as n/a.
The RFC is under review : opea-project/docs#49

Type of change

List the type of change like below. Please delete options that are not relevant.

New feature (non-breaking change which adds new functionality)

Dependencies

List the newly introduced 3rd party dependency if exists.

Tests

Describe the tests that you ran to verify your changes.

Embedding

codecov · 2024-08-20T22:26:55Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Files	Coverage Δ
comps/cores/proto/docarray.py	`98.95% <100.00%> (+0.01%)`	⬆️

... and 1 file with indirect coverage changes

srinarayan-srikanthan · 2024-08-20T22:37:56Z

@chensuyue can you please tag this PR with 'V0.9' Milestone

Signed-off-by: srinarayan-srikanthan <[email protected]>

Signed-off-by: zepan <[email protected]>

Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: NeuralChatBot <[email protected]>

Signed-off-by: zepan <[email protected]>

* refine test script of embedding llama_index Signed-off-by: letonghan <[email protected]> * update Signed-off-by: letonghan <[email protected]> * update code of llm-native Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix llm native issue Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * udpate param & add readme Signed-off-by: letonghan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: zepan <[email protected]>

Signed-off-by: Sun, Xuehao <[email protected]>

…hing docker (opea-project#438) Signed-off-by: Wang, Kai Lawrence <[email protected]>

Signed-off-by: zepan <[email protected]>

Co-authored-by: Abolfazl Shahbazi <[email protected]>

Only one H1 heading (first for the title) allowed. Signed-off-by: David B. Kinder <[email protected]> Co-authored-by: chen, suyue <[email protected]>

Signed-off-by: lvliang-intel <[email protected]>

* update wf Signed-off-by: chensuyue <[email protected]> * for test Signed-off-by: chensuyue <[email protected]> * remove test code Signed-off-by: chensuyue <[email protected]> --------- Signed-off-by: chensuyue <[email protected]>

…punctuation mark location (opea-project#513) * Revise SpeechT5Model to end the last audio chunk at the correct punctuation mark Signed-off-by: Chun Tao <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * still append the last chunk if it contains no punctuation Signed-off-by: Chun Tao <[email protected]> --------- Signed-off-by: Chun Tao <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: letonghan <[email protected]>

Signed-off-by: Sun, Xuehao <[email protected]>

* set hf_hub to 0.24.0 * add docgrader to agent strategy openai llm code passed * add nonstreaming output for agent * add react langgraph and tests * fix react langchain bug Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test script Signed-off-by: minmin-intel <[email protected]> * fix bug in test script Signed-off-by: minmin-intel <[email protected]> * update readme and rm old agentic-rag strategy * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update test and docgrader readme * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix bug in test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update test * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update rag agent strategy name and update readme Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update test Signed-off-by: minmin-intel <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: minmin-intel <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

…pea-project#314) * Add Retrieval gateway Signed-off-by: Chendi.Xue <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update gateway to latest protocal Signed-off-by: Chendi.Xue <[email protected]> * tested with DocIndexer, rebased gateway is now workable Signed-off-by: Chendi.Xue <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: Chendi.Xue <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: letonghan <[email protected]>

Signed-off-by: Abolfazl Shahbazi <[email protected]>

Signed-off-by: Lianhao Lu <[email protected]>

Signed-off-by: letonghan <[email protected]>

Signed-off-by: Xinyao Wang <[email protected]>

Signed-off-by: Abolfazl Shahbazi <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * for test. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add func. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * for test. Signed-off-by: zepan <[email protected]> * For test2. Signed-off-by: zepan <[email protected]> * add function. Signed-off-by: zepan <[email protected]> * Modify get matrix. Signed-off-by: zepan <[email protected]> * modify get matrix. Signed-off-by: zepan <[email protected]> * modify input. Signed-off-by: zepan <[email protected]> * Add comps workflow. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add node. Signed-off-by: zepan <[email protected]> * Add images build. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue node. Signed-off-by: zepan <[email protected]> * Add images build. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue of yaml name. Signed-off-by: zepan <[email protected]> * Fix issue of service value. Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * restore build Image. Signed-off-by: zepan <[email protected]> * modify Build Image. Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * modify Signed-off-by: zepan <[email protected]> * modify call service. Signed-off-by: zepan <[email protected]> * Fix issue work_dir Signed-off-by: zepan <[email protected]> * Fix issue docker_compose_path Signed-off-by: zepan <[email protected]> * Fix tag. Signed-off-by: zepan <[email protected]> * Change Work_dir. Signed-off-by: zepan <[email protected]> * Change image-build branch Signed-off-by: zepan <[email protected]> * Change work_dir. Signed-off-by: zepan <[email protected]> * Chaange work_dir and image_build branch Signed-off-by: zepan <[email protected]> * Add test-service-compose. Signed-off-by: zepan <[email protected]> * Remove Unrelated variables. Signed-off-by: zepan <[email protected]> * for test Signed-off-by: zepan <[email protected]> * test2 Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * for test. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add func. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * for test. Signed-off-by: zepan <[email protected]> * For test2. Signed-off-by: zepan <[email protected]> * add function. Signed-off-by: zepan <[email protected]> * Modify get matrix. Signed-off-by: zepan <[email protected]> * modify get matrix. Signed-off-by: zepan <[email protected]> * modify input. Signed-off-by: zepan <[email protected]> * Add comps workflow. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add node. Signed-off-by: zepan <[email protected]> * Add images build. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue node. Signed-off-by: zepan <[email protected]> * Add images build. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue of yaml name. Signed-off-by: zepan <[email protected]> * Fix issue of service value. Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * restore build Image. Signed-off-by: zepan <[email protected]> * modify Build Image. Signed-off-by: zepan <[email protected]> * test Signed-off-by: zepan <[email protected]> * modify Signed-off-by: zepan <[email protected]> * modify call service. Signed-off-by: zepan <[email protected]> * Fix issue work_dir Signed-off-by: zepan <[email protected]> * Fix issue docker_compose_path Signed-off-by: zepan <[email protected]> * Fix tag. Signed-off-by: zepan <[email protected]> * Change Work_dir. Signed-off-by: zepan <[email protected]> * Change image-build branch Signed-off-by: zepan <[email protected]> * Change work_dir. Signed-off-by: zepan <[email protected]> * Chaange work_dir and image_build branch Signed-off-by: zepan <[email protected]> * Add test-service-compose. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove Unrelated variables. Signed-off-by: zepan <[email protected]> * for test Signed-off-by: zepan <[email protected]> * test2 Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue of run-test not be run. Signed-off-by: zepan <[email protected]> * for test Signed-off-by: zepan <[email protected]> * Change input. Signed-off-by: zepan <[email protected]> * Fix issue of input.service Signed-off-by: zepan <[email protected]> * Fix output issue. Signed-off-by: zepan <[email protected]> * Fix " issue. Signed-off-by: zepan <[email protected]> * Change the trigger method to manual. Signed-off-by: zepan <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix issue of run-test not be run. Signed-off-by: zepan <[email protected]> * for test Signed-off-by: zepan <[email protected]> * Change input. Signed-off-by: zepan <[email protected]> * Fix issue of input.service Signed-off-by: zepan <[email protected]> * Fix output issue. Signed-off-by: zepan <[email protected]> * Fix " issue. Signed-off-by: zepan <[email protected]> * Change the trigger method to manual. Signed-off-by: zepan <[email protected]> * Rename yaml. Signed-off-by: zepan <[email protected]> --------- Signed-off-by: zepan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

for more information, see https://pre-commit.ci

BaoHuiling · 2024-08-21T07:31:33Z

Replaced this PR by #538 to fix DCO error

NeoZhangJianyu and others added 13 commits August 13, 2024 13:20

embedding branch

e973b46

Updated input and output

aa32eb4

Update embeddings_clip.py

06924d0

Merge pull request #5 from srinarayan-srikanthan/embedding

8e2bcac

Embedding

Update embedding_multimodal.py

c036ddc

Update embeddings_clip.py

ac47c5b

Update embedding_multimodal.py

716a4b7

Typo fix

6096cf5

Typo fix

3751c67

Update embeddings_clip.py

14e30f0

Update embeddings_clip.py

5325e69

Added readme

4775b9b

Added torchvision

3917030

srinarayan-srikanthan requested review from XuhuiRen and lvliang-intel as code owners August 20, 2024 22:11

srinarayan-srikanthan and others added 13 commits August 20, 2024 20:34

Update README.md

30aa686

Signed-off-by: srinarayan-srikanthan <[email protected]>

Update requirements.txt

ca2b4b0

Signed-off-by: srinarayan-srikanthan <[email protected]>

Added constraints

30e99c0

Signed-off-by: srinarayan-srikanthan <[email protected]>

Update embedding_multimodal.py

afe19bc

Signed-off-by: srinarayan-srikanthan <[email protected]>

Update docarray.py

b8b86c6

Signed-off-by: srinarayan-srikanthan <[email protected]>

Check build fail and change port 8008 to 5025/5026. (opea-project#487)

8533974

Signed-off-by: zepan <[email protected]>

Freeze requirements workflow for release tag (opea-project#485)

dd2d024

Signed-off-by: Sun, Xuehao <[email protected]> Signed-off-by: NeuralChatBot <[email protected]>

Standardize image build. (opea-project#472)

9c307de

Signed-off-by: zepan <[email protected]>

Add validate result detection. (opea-project#492)

1174bb5

Signed-off-by: zepan <[email protected]>

Modify validate result check. (opea-project#503)

37655b8

Signed-off-by: zepan <[email protected]>

Fix line endings to LF (opea-project#497)

23a21b2

Signed-off-by: Sun, Xuehao <[email protected]>

fix asr ut (opea-project#501)

f0e4656

wangkl2 and others added 24 commits August 20, 2024 20:34

Add cmds to restart ollama service and add proxy settings while launc…

b704251

…hing docker (opea-project#438) Signed-off-by: Wang, Kai Lawrence <[email protected]>

Fix waring. (opea-project#504)

6c9f382

Signed-off-by: zepan <[email protected]>

Update README.md (opea-project#482)

f6f8445

Co-authored-by: Abolfazl Shahbazi <[email protected]>

doc: fix multiple H1 headings (opea-project#481)

7ed94b3

Only one H1 heading (first for the title) allowed. Signed-off-by: David B. Kinder <[email protected]> Co-authored-by: chen, suyue <[email protected]>

Remove unnecessary langsmith dependency (opea-project#369)

fc47d17

Signed-off-by: lvliang-intel <[email protected]>

Add Nginx Component for Service Forwarding (opea-project#506)

a3b119d

Signed-off-by: letonghan <[email protected]>

Freeze base images (opea-project#509)

ec9364f

Signed-off-by: Sun, Xuehao <[email protected]>

Refine Nginx Component (opea-project#523)

0508339

Signed-off-by: letonghan <[email protected]>

adding entrypoint.sh to faq-generation comp (opea-project#525)

998db33

Signed-off-by: Abolfazl Shahbazi <[email protected]>

Fix tts image push error (opea-project#530)

578474f

Signed-off-by: Lianhao Lu <[email protected]>

refine llm parameter (opea-project#532)

2be12d3

Signed-off-by: letonghan <[email protected]>

Add logging for unified debug (opea-project#521)

07add19

Signed-off-by: Xinyao Wang <[email protected]>

Using habana docker 1.16.1 everywhere (opea-project#528)

1c0c8b5

Signed-off-by: Abolfazl Shahbazi <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

427f12a

for more information, see https://pre-commit.ci

Update __init__.py

d58f542

Update embeddings_clip.py

1cf05f4

[pre-commit.ci] auto fixes from pre-commit.com hooks

13f613f

for more information, see https://pre-commit.ci

Update embedding_multimodal.py

867a1a3

[pre-commit.ci] auto fixes from pre-commit.com hooks

994b43a

for more information, see https://pre-commit.ci

srinarayan-srikanthan force-pushed the sri-embedding branch from f64b1be to 994b43a Compare August 21, 2024 03:39

srinarayan-srikanthan requested review from Spycsh, letonghan and XinyuYe-Intel as code owners August 21, 2024 03:39

srinarayan-srikanthan closed this Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding embedding support for CLIP based models for VideoRAGQnA #533

adding embedding support for CLIP based models for VideoRAGQnA #533

srinarayan-srikanthan commented Aug 20, 2024

codecov bot commented Aug 20, 2024 •

edited

Loading

srinarayan-srikanthan commented Aug 20, 2024

BaoHuiling commented Aug 21, 2024 •

edited

Loading

adding embedding support for CLIP based models for VideoRAGQnA #533

adding embedding support for CLIP based models for VideoRAGQnA #533

Conversation

srinarayan-srikanthan commented Aug 20, 2024

Description

Issues

Type of change

Dependencies

Tests

codecov bot commented Aug 20, 2024 • edited Loading

Codecov Report

srinarayan-srikanthan commented Aug 20, 2024

BaoHuiling commented Aug 21, 2024 • edited Loading

codecov bot commented Aug 20, 2024 •

edited

Loading

BaoHuiling commented Aug 21, 2024 •

edited

Loading