Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add MilvusStore and re-implement MapStore and ChromadbStore, support multi index for one store #322

Merged
merged 66 commits into from
Nov 8, 2024
Merged

Add MilvusStore and re-implement MapStore and ChromadbStore, support multi index for one store #322

merged 66 commits into from
Nov 8, 2024

Conversation

ouonline
Copy link
Contributor

@ouonline ouonline commented Oct 24, 2024 •
edited
Loading

  1. add IndexBase and StoreBase
  2. add MilvusStore and re-implement MapStore and ChromadbStore
  3. users can specify store and index types for Document
  4. similarity functions are registered to global

@ouonline ouonline force-pushed the index-api branch 6 times, most recently from 336f986 to 5d56948 Compare October 25, 2024 01:18
@ouonline ouonline changed the title [skip ci]Index api [skip ci] breaking changes of store and index apis Oct 28, 2024
@ouonline ouonline force-pushed the index-api branch 3 times, most recently from 1c11e33 to 1eae574 Compare October 28, 2024 11:39
wzh1994
wzh1994 reviewed Oct 29, 2024
View reviewed changes
lazyllm/tools/rag/__init__.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/base_index.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/base_store.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/index.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/doc_impl.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/base_store.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/doc_impl.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/store.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/doc_impl.py Show resolved Hide resolved
lazyllm/tools/rag/index.py Outdated Show resolved Hide resolved
wzh1994
wzh1994 reviewed Nov 5, 2024
View reviewed changes
lazyllm/tools/rag/smart_embedding_index.py

@override
def update(self, nodes: List[DocNode]) -> None:
self._store.update_nodes(nodes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

按照之前沟通的，这里后面可以优化成，丢掉一些非关键信息进行存储，召回后再从id恢复出全局信息

@ouonline ouonline changed the title [skip ci] breaking changes of store and index apis breaking changes of store and index apis Nov 6, 2024
@ouonline ouonline force-pushed the index-api branch 2 times, most recently from f657a46 to 1c4caa9 Compare November 6, 2024 03:17
wzh1994
wzh1994 reviewed Nov 6, 2024
View reviewed changes
lazyllm/tools/rag/doc_impl.py Outdated Show resolved Hide resolved
lazyllm/tools/rag/doc_impl.py Outdated Show resolved Hide resolved
@ouonline ouonline force-pushed the index-api branch 3 times, most recently from 5a3bba6 to bc2cb1b Compare November 6, 2024 08:46
@ouonline ouonline force-pushed the index-api branch 4 times, most recently from aaeee94 to 3563d09 Compare November 7, 2024 11:59
@ouonline ouonline marked this pull request as ready for review November 7, 2024 13:12
@wzh1994 wzh1994 changed the title breaking changes of store and index apis Add MilvusStore and re-implement MapStore and ChromadbStore, support multi index for one store Nov 8, 2024
@wzh1994 wzh1994 merged commit 5d8a157 into LazyAGI:main Nov 8, 2024
11 checks passed
@ouonline ouonline deleted the index-api branch November 8, 2024 06:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wzh1994 wzh1994 wzh1994 left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ouonline @wzh1994 @lwj-st