Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add VideoRAGQnA as MMRAG usecase in Example #744

Merged
merged 34 commits into from
Sep 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
7c7eada
Initial commit
BaoHuiling Aug 19, 2024
8622376
refactor ui
BaoHuiling Aug 20, 2024
2eae251
save work
BaoHuiling Aug 30, 2024
9190a6c
save work
BaoHuiling Aug 30, 2024
bd96334
move to sub-folder
BaoHuiling Aug 30, 2024
a74e33e
Merge branch 'opea-project:main' into example-dev
BaoHuiling Sep 2, 2024
e5ff19b
save
BaoHuiling Sep 2, 2024
ed36fff
workable with fake microservice
BaoHuiling Sep 3, 2024
3f741a1
Merge branch 'opea-project:main' into example-dev
BaoHuiling Sep 4, 2024
8404b1f
update dataprep
BaoHuiling Sep 4, 2024
f5842ce
update dataprep
BaoHuiling Sep 5, 2024
2d32fd1
Merge branch 'opea-project:main' into example-dev
BaoHuiling Sep 5, 2024
4019931
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 5, 2024
7d8e334
add test script
BaoHuiling Sep 6, 2024
77648ea
Merge branch 'main' into example-dev
kevinintel Sep 6, 2024
808e434
mv test script
BaoHuiling Sep 6, 2024
d8b4ea6
add readme
BaoHuiling Sep 6, 2024
692dbd7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 6, 2024
ec43c8e
rm type
BaoHuiling Sep 6, 2024
ca9499c
add insertion after dataprep
BaoHuiling Sep 7, 2024
af7f581
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Sep 7, 2024
7462278
rename script
BaoHuiling Sep 10, 2024
094b8ea
Merge branch 'main' into example-dev
BaoHuiling Sep 10, 2024
5bac304
rm upper folder
BaoHuiling Sep 10, 2024
c8045b3
Merge branch 'main' into example-dev
kevinintel Sep 10, 2024
db54560
Merge branch 'main' into example-dev
BaoHuiling Sep 10, 2024
43990ea
update file structure
BaoHuiling Sep 11, 2024
17e7439
Merge branch 'main' into example-dev
BaoHuiling Sep 11, 2024
5e10972
Merge branch 'main' into example-dev
BaoHuiling Sep 11, 2024
91eefd7
use repo/tag in compose
BaoHuiling Sep 11, 2024
7308218
rename image
BaoHuiling Sep 11, 2024
6f26119
update name in script
BaoHuiling Sep 11, 2024
e6c8033
update readme
BaoHuiling Sep 11, 2024
565f82d
Merge branch 'main' into example-dev
BaoHuiling Sep 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions VideoRAGQnA/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@


# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

FROM python:3.11-slim

RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
libgl1-mesa-glx \
libjemalloc-dev \
git

RUN useradd -m -s /bin/bash user && \
mkdir -p /home/user && \
chown -R user /home/user/

WORKDIR /home/user/

RUN git clone https://github.com/opea-project/GenAIComps.git

WORKDIR /home/user/GenAIComps
RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir -r /home/user/GenAIComps/requirements.txt

COPY ./videoragqna.py /home/user/videoragqna.py

ENV PYTHONPATH=$PYTHONPATH:/home/user/GenAIComps

USER user

WORKDIR /home/user

ENTRYPOINT ["python", "videoragqna.py"]
107 changes: 107 additions & 0 deletions VideoRAGQnA/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# VideoRAGQnA Application

Video RAG QnA is a framework that retrieves video based on provided user prompt. It uses only the video embeddings to perform vector similarity search in Intel's VDMS vector database and performs all operations on Intel Xeon CPU. The pipeline supports long form videos and time-based search.

VideoRAGQnA is implemented on top of [GenAIComps](https://github.com/opea-project/GenAIComps), with the architecture flow chart shows below:

```mermaid
---
config:
flowchart:
nodeSpacing: 100
rankSpacing: 100
curve: linear
theme: base
themeVariables:
fontSize: 42px
---
flowchart LR
%% Colors %%
classDef blue fill:#ADD8E6,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5
classDef orange fill:#FBAA60,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5
classDef orchid fill:#C26DBC,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5
classDef invisible fill:transparent,stroke:transparent;
style VideoRAGQnA-MegaService stroke:#000000
%% Subgraphs %%
subgraph VideoRAGQnA-MegaService["VideoRAGQnA-MegaService"]
direction LR
EM([Embedding <br>]):::blue
RET([Retrieval <br>]):::blue
RER([Rerank <br>]):::blue
LLM([LLM <br>]):::blue
end
subgraph User Interface
direction TB
a([User Input Query]):::orchid
UI([UI server<br>]):::orchid
Ingest([Ingest<br>]):::orchid
end
subgraph VideoRAGQnA GateWay
direction LR
invisible1[ ]:::invisible
GW([VideoRAGQnA GateWay<br>]):::orange
end
subgraph .
X([OPEA Micsrservice]):::blue
Y{{Open Source Service}}
Z([OPEA Gateway]):::orange
Z1([UI]):::orchid
end

LOCAL_RER{{Reranking service<br>}}
CLIP_EM{{Embedding service <br>}}
VDB{{Vector DB<br><br>}}
V_RET{{Retriever service <br>}}
Ingest{{Ingest data <br>}}
DP([Data Preparation<br>]):::blue
LVM_gen{{LLM Service <br>}}

%% Data Preparation flow
%% Ingest data flow
direction LR
Ingest[Ingest data] -->|a| DP
DP <-.->|b| CLIP_EM

%% Questions interaction
direction LR
a[User Input Query] -->|1| UI
UI -->|2| GW
GW <==>|3| VideoRAGQnA-MegaService
EM ==>|4| RET
RET ==>|5| RER
RER ==>|6| LLM


%% Embedding service flow
direction TB
EM <-.->|3'| CLIP_EM
RET <-.->|4'| V_RET
RER <-.->|5'| LOCAL_RER
LLM <-.->|6'| LVM_gen

direction TB
%% Vector DB interaction
V_RET <-.->|d|VDB
DP <-.->|d|VDB
```

This VideoRAGQnA use case performs RAG using LangChain, Intel VDMS VectorDB and Text Generation Inference on Intel XEON Scalable Processors.

## Deploy VideoRAGQnA Service

The VideoRAGQnA service can be effortlessly deployed on Intel XEON Scalable Processors.

### Required Models

By default, the embedding and LVM models are set to a default value as listed below:

| Service | Model |
| --------- | ---------------------------- |
| Embedding | openai/clip-vit-base-patch32 |
| LVM | DAMO-NLP-SG/Video-LLaMA |

### Deploy VideoRAGQnA on Xeon

For full instruction of deployment, please check [Guide](docker/xeon/README.md)

Currently we support deploying VideoRAGQnA services with docker compose, using the docker images `built from source`. Find the corresponding [compose.yaml](./docker/xeon/compose.yaml).
Binary file added VideoRAGQnA/assets/img/video-rag-qna.gif
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading