Multimodal dataprep

[tileintel]

Description

This PR introduces multimodal dataprep microservice. This microservice is required for Multimodal RAG on Videos application. This allows users to upload mp4 videos and their associated transcripts (optional) and ingests them into Redis vector store.

This microservice provides 3 different API allowing users to upload and ingest videos for 3 use cases:

1. when a transcript file (under .vtt format) is available for each video.
2. when a video has meaningful audio or recognizable speech but its transcript file is not available. In this use case, this microservice will use whisper model to generate the .vtt transcript for the video.
3. when a video does not have meaningful audio or does not have audio. In this use case, transcript either does not provide any meaningful information or does not exist. Thus, it is preferred to leverage a LVM microservice to summarize the video frames.

This microservice also provides an API for user to list all videos ingested under current index name and an API to delete all videos from local storage and from redis vector store under current index name.

Issues

RFC: https://github.com/opea-project/docs/pull/49/files
Issue: opea-project/GenAIExamples#358

Type of change

List the type of change like below. Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• [x ] New feature (non-breaking change which adds new functionality)
• Breaking change (fix or feature that would break existing design and interface)
• Others (enhancement, documentation, validation, etc.)

Dependencies

docarray[full]
fastapi
langchain==0.1.12
langchain_benchmarks
langsmith
moviepy
opencv-python
openai-whisper
opentelemetry-api
opentelemetry-exporter-otlp
opentelemetry-sdk
Pillow
prometheus-fastapi-instrumentator
pydantic==2.8.2
python-multipart
redis
transformers
shortuuid
uvicorn
webvtt-py
llava-hf/llava-1.5-7b-hf

Tests

We have provided 1 test for this microservice.

• tests/test_dataprep_redis_multimodal_langchain.sh: This test downloads a video from here and prepares a transcription for the downloaded video. Then it uses them to test the 3 provided API. This test also tests get_videos and delete_videos APIs.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines	Coverage Δ
comps/cores/proto/docarray.py	99.09% <100.00%> (+0.08%)	⬆️

... and 1 file with indirect coverage changes

[tileintel]

@lvliang-intel. Here is another PR for multimodal data prep with Redis for Multimodal RAG. This PR includes changes in the PR #555 because #555 has not been reviewed and merged yet to GenAIComps main branch.
Would you please help to review this PR?
cc: @kevinintel

[XuhuiRen]

should be better to maintain a single PR, considering merge #555 to this one?
also, the suggestions in that PR should moved into this PR

[tileintel]

should be better to maintain a single PR, considering merge #555 to this one? also, the suggestions in that PR should moved into this PR

Thank @XuhuiRen for your suggestion. I have merged the PR #555 to this one. I cannot move the suggestions in PR #555 here. However, I have addressed all of the conversations in #555 into this PR.
I apologize for any inconvenience. I originally thought that separating different microservice into different PRs will help reviewers. But it seems not.

[XuhuiRen]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

[XuhuiRen]

should be better to maintain a single PR, considering merge #555 to this one? also, the suggestions in that PR should moved into this PR

Thank @XuhuiRen for your suggestion. I have merged the PR #555 to this one. I cannot move the suggestions in PR #555 here. However, I have addressed all of the conversations in #555 into this PR. I apologize for any inconvenience. I originally thought that separating different microservice into different PRs will help reviewers. But it seems not.

i recommend to add the code for image embedding to maintain a comprehensive service.

[tileintel]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

You are correct. We haven't submitted the retriever microservice yet. We are waiting for #555 and #575 merged first before we submit the last PRs for retrieval and lvm.
Also please note that we have several GenAIExamples as well. For multimodal rag on videos we will not make use of rerank microservice.

[BaoHuiling]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

hi Xuhui. To clarify this, we are contributing different use case for MMRAG, and I’m working on VideoRAGQnA, which is PR #495, #496, #538, #539 and we are going to contribute another PR for dataprep

[XuhuiRen]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

hi Xuhui. To clarify this, we are contributing different use case for MMRAG, and I’m working on VideoRAGQnA, which is PR #495, #496, #538, #539 and we are going to contribute another PR for dataprep

but it seems that PR #538 has name conflict with this PR?

[tileintel]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

hi Xuhui. To clarify this, we are contributing different use case for MMRAG, and I’m working on VideoRAGQnA, which is PR #495, #496, #538, #539 and we are going to contribute another PR for dataprep

but it seems that PR #538 has name conflict with this PR?

Yes. #538 was developed from a previous commit of this PR. I would suggest that we review and merge #575 first, and @BaoHuiling and I will resolve the conflict from #538 right after. I have mentioned this in the related Issue-538

[BaoHuiling]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

hi Xuhui. To clarify this, we are contributing different use case for MMRAG, and I’m working on VideoRAGQnA, which is PR #495, #496, #538, #539 and we are going to contribute another PR for dataprep

but it seems that PR #538 has name conflict with this PR?

yes there should be some conflicts, we will align it tomorrow and please hold those PR until we resolve the conflicts. thanks!

[BaoHuiling]

@srinarayan-srikanthan @s-gobriel please take a look on this PR, check the conflicts and we need to update the code

[BaoHuiling]

@BaoHuiling @tileintel i did not see the retriever for multimodal. there is only a reranker PR. Is there any PR I missed?

hi Xuhui. To clarify this, we are contributing different use case for MMRAG, and I’m working on VideoRAGQnA, which is PR #495, #496, #538, #539 and we are going to contribute another PR for dataprep

but it seems that PR #538 has name conflict with this PR?

Yes. #538 was developed from a previous commit of this PR. I would suggest that we review and merge #575 first, and @BaoHuiling and I will resolve the conflict from #538 right after. I have mentioned this in the related Issue-538

I'm okay with this. Please do it

[tileintel]

@XuhuiRen Given that, @XuhuiRen and @lvliang-intel already approved this PR, would you please help to merge this? This will help us to resolve conflicts for another related PR quicker. Thanks