-
Notifications
You must be signed in to change notification settings - Fork 144
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add: Pathway vector store and retriever as LangChain component #342
Conversation
7cd80aa
to
5e80ced
Compare
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
…n components Signed-off-by: Berke <[email protected]>
for more information, see https://pre-commit.ci
fix: update docker to include libmagic Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
for more information, see https://pre-commit.ci
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @berkecanrizai for the contribution.
Most of my change requests are repeated within the docs, so it should be easy to make the changes. However if you need any help, please feel free to post here.
Thanks.
…proxy. Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
for more information, see https://pre-commit.ci
Hey, I think I resolved your comments above, thanks for the clear instructions! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @berkecanrizai
Nice work and LGTM!
Please make sure all other reviewers' feedbacks are resolved before merging.
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
Signed-off-by: Berke <[email protected]>
…project#342) * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * docker Signed-off-by: Berke <[email protected]> * example data Signed-off-by: Berke <[email protected]> * docs(readme): update, add commands Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused parts Signed-off-by: Berke <[email protected]> * fix: image name, compose env vars Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * fix: logging name Signed-off-by: Berke <[email protected]> * fix: env var Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * docs(readme): update input sources Signed-off-by: Berke <[email protected]> * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: mv vector store, naming, clarify instructions, improve ingestion components Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: add pw retriever test fix: update docker to include libmagic Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement suggestions from review, entrypoint, reqs, comments, https_proxy. Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: update docker tags in test and readme Signed-off-by: Berke <[email protected]> * tests: add separate pathway vectorstore test Signed-off-by: Berke <[email protected]> --------- Signed-off-by: Berke <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]>
…project#342) * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * docker Signed-off-by: Berke <[email protected]> * example data Signed-off-by: Berke <[email protected]> * docs(readme): update, add commands Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused parts Signed-off-by: Berke <[email protected]> * fix: image name, compose env vars Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * fix: logging name Signed-off-by: Berke <[email protected]> * fix: env var Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * docs(readme): update input sources Signed-off-by: Berke <[email protected]> * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: mv vector store, naming, clarify instructions, improve ingestion components Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: add pw retriever test fix: update docker to include libmagic Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement suggestions from review, entrypoint, reqs, comments, https_proxy. Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: update docker tags in test and readme Signed-off-by: Berke <[email protected]> * tests: add separate pathway vectorstore test Signed-off-by: Berke <[email protected]> --------- Signed-off-by: Berke <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]>
* add rerank with neural speed Signed-off-by: Dong, Bo1 <[email protected]> * add the code Signed-off-by: Dong, Bo1 <[email protected]> * add the code Signed-off-by: Dong, Bo1 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Dong, Bo1 <[email protected]> * fix mismatched response format w/wo streaming guardrails (#568) * fix mismatched response format w/wo streaming guardrails * fix & debug * fix & rm debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * Fix guardrails out handle logics for space linebreak and quote (#571) * fix mismatched response format w/wo streaming guardrails * fix & debug * fix & rm debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * debug * debug * debug * fix pre-space and linebreak * fix pre-space and linebreak * fix single/double quote * fix single/double quote * remove debug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * BUG FIX: LVM security fix (#572) * add url validator Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add validation for video_url Signed-off-by: BaoHuiling <[email protected]> --------- Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * Modify output messages. (#569) * Reduced output. Signed-off-by: zepan <[email protected]> * Output the location where the modified Dockerfile file is referenced. Signed-off-by: zepan <[email protected]> * for test Signed-off-by: zepan <[email protected]> * Restore test file. Signed-off-by: zepan <[email protected]> --------- Signed-off-by: zepan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * refine logging code. (#559) * add ut and refine logging code. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update microservice port. --------- Co-authored-by: root <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * adding lancedb to langchain vectorstores (#291) * adding lancedb to langchain vectorstores Signed-off-by: sharanshirodkar7 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: sharanshirodkar7 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: lvliang-intel <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * Refine Dataprep Milvus MS (#570) Signed-off-by: letonghan <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * final version Signed-off-by: Dong, Bo1 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Dong, Bo1 <[email protected]> * update the readme Signed-off-by: Dong, Bo1 <[email protected]> * add the sign Signed-off-by: Dong, Bo1 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Dong, Bo1 <[email protected]> * fix error for pre ci Signed-off-by: Dong, Bo1 <[email protected]> * add the ut Signed-off-by: Dong, Bo1 <[email protected]> * update docker file Signed-off-by: Dong, Bo1 <[email protected]> * update CI test log achieve (#577) Signed-off-by: chensuyue <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * Multimodal dataprep (#575) * multimodal embedding for MM RAG for videos Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * develop data prep first commit Signed-off-by: Tiep Le <[email protected]> * develop dataprep microservice for multimodal data Signed-off-by: Tiep Le <[email protected]> * multimodal langchain for dataprep Signed-off-by: Tiep Le <[email protected]> * update README Signed-off-by: Tiep Le <[email protected]> * update README Signed-off-by: Tiep Le <[email protected]> * update README Signed-off-by: Tiep Le <[email protected]> * update README Signed-off-by: Tiep Le <[email protected]> * cosmetic Signed-off-by: Tiep Le <[email protected]> * test for multimodal dataprep Signed-off-by: Tiep Le <[email protected]> * update test Signed-off-by: Tiep Le <[email protected]> * update test Signed-off-by: Tiep Le <[email protected]> * update test Signed-off-by: Tiep Le <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * cosmetic update Signed-off-by: Tiep Le <[email protected]> * remove langsmith Signed-off-by: Tiep Le <[email protected]> * update API to remove /dataprep from API names and remove langsmith Signed-off-by: Tiep Le <[email protected]> * update test Signed-off-by: Tiep Le <[email protected]> * update the error message per PR reviewer Signed-off-by: Tiep Le <[email protected]> --------- Signed-off-by: Tiep Le <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * add: Pathway vector store and retriever as LangChain component (#342) * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * docker Signed-off-by: Berke <[email protected]> * example data Signed-off-by: Berke <[email protected]> * docs(readme): update, add commands Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused parts Signed-off-by: Berke <[email protected]> * fix: image name, compose env vars Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * fix: logging name Signed-off-by: Berke <[email protected]> * fix: env var Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * docs(readme): update input sources Signed-off-by: Berke <[email protected]> * nb Signed-off-by: Berke <[email protected]> * init changes Signed-off-by: Berke <[email protected]> * fix: formatting, data sources Signed-off-by: Berke <[email protected]> * docs(readme): update instructions, add comments Signed-off-by: Berke <[email protected]> * fix: rm unused part Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * fix: rename pw docker Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * feat: mv vector store, naming, clarify instructions, improve ingestion components Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * tests: add pw retriever test fix: update docker to include libmagic Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * implement suggestions from review, entrypoint, reqs, comments, https_proxy. Signed-off-by: Berke <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix: update docker tags in test and readme Signed-off-by: Berke <[email protected]> * tests: add separate pathway vectorstore test Signed-off-by: Berke <[email protected]> --------- Signed-off-by: Berke <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * Add local Rerank microservice for VideoRAGQnA (#496) * initial commit Signed-off-by: BaoHuiling <[email protected]> * save Signed-off-by: BaoHuiling <[email protected]> * add readme, test script, fix bug Signed-off-by: BaoHuiling <[email protected]> * update video URL Signed-off-by: BaoHuiling <[email protected]> * use default Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update core dependency Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * use p 5000 Signed-off-by: BaoHuiling <[email protected]> * use 5037 Signed-off-by: BaoHuiling <[email protected]> * update ctnr name Signed-off-by: BaoHuiling <[email protected]> * remove langsmith Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add rerank algo desc in readme Signed-off-by: BaoHuiling <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Signed-off-by: BaoHuiling <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: chen, suyue <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * Add Scan Container. (#560) Signed-off-by: zepan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * fix SearchedMultimodalDoc in docarray (#583) Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * update image build yaml (#529) Signed-off-by: chensuyue <[email protected]> Signed-off-by: zepan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * add microservice for intent detection (#131) * add microservice for intent detection Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update license copyright Signed-off-by: Liangyx2 <[email protected]> * add ut Signed-off-by: Liangyx2 <[email protected]> * refine Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update folder Signed-off-by: Liangyx2 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix test Signed-off-by: Liangyx2 <[email protected]> --------- Signed-off-by: Liangyx2 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Signed-off-by: Dong, Bo1 <[email protected]> * Make the scanning method optional. (#580) Signed-off-by: zepan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * add code owners (#586) Signed-off-by: Dong, Bo1 <[email protected]> * remove revision for tei (#584) Signed-off-by: letonghan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * Bug fix (#591) * Check if the document exists. Signed-off-by: zepan <[email protected]> * Add flag output. Signed-off-by: zepan <[email protected]> * Modify nginx readme. Signed-off-by: zepan <[email protected]> * Modify document detection logic Signed-off-by: zepan <[email protected]> --------- Signed-off-by: zepan <[email protected]> Signed-off-by: Dong, Bo1 <[email protected]> * fix ut issue Signed-off-by: Dong, Bo1 <[email protected]> * merge the main Signed-off-by: Dong, Bo1 <[email protected]> * align with new pipeline Signed-off-by: Dong, Bo1 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * align with newest pipeline Signed-off-by: Dong, Bo1 <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * upload code Signed-off-by: Dong, Bo1 <[email protected]> * update the ut Signed-off-by: Dong, Bo1 <[email protected]> * add docker path Signed-off-by: Dong, Bo1 <[email protected]> * add the docker path Signed-off-by: Dong, Bo1 <[email protected]> --------- Signed-off-by: Dong, Bo1 <[email protected]> Signed-off-by: BaoHuiling <[email protected]> Signed-off-by: zepan <[email protected]> Signed-off-by: sharanshirodkar7 <[email protected]> Signed-off-by: letonghan <[email protected]> Signed-off-by: chensuyue <[email protected]> Signed-off-by: Tiep Le <[email protected]> Signed-off-by: Berke <[email protected]> Signed-off-by: Liangyx2 <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Sihan Chen <[email protected]> Co-authored-by: Huiling Bao <[email protected]> Co-authored-by: ZePan110 <[email protected]> Co-authored-by: lkk <[email protected]> Co-authored-by: root <[email protected]> Co-authored-by: Sharan Shirodkar <[email protected]> Co-authored-by: lvliang-intel <[email protected]> Co-authored-by: Letong Han <[email protected]> Co-authored-by: chen, suyue <[email protected]> Co-authored-by: Tiep Le <[email protected]> Co-authored-by: berkecanrizai <[email protected]> Co-authored-by: Liangyx2 <[email protected]> Co-authored-by: kevinintel <[email protected]>
Description
Adds Pathway Vector Store as a vector DB option.
Note that Pathway expects text documents for the retrieval, thus I added the embedder (both the local HF model and the
HuggingFaceHubEmbeddings(model=tei_embedding_endpoint)
) to the vector store initialization.Another additional note, Pathway connects to the input sources from the start, and does not allow
upsert
type operations via client. For now, I set the data source as the localdata
folder. It is possible to use any other data source, check out here.Issues
Type of change
List the type of change like below. Please delete options that are not relevant.
Pathway vector DB and retriever.
Dependencies
Pathway is needed if server is being started outside of the Docker.
Retriever and the other services do not need any change in the requirements.
Tests