RAG Enhancement

[wzh1994]

• 1. register DataReader with rule which specifies file name or file type
  • Add rag readers for replace llamaindex #239
• 2. add file manager to manage document sources, e.g. nfs, ceph, minio, etc.
• 3. use scan instead of notify for file changes
  • RAG: use scan instead of notify for file changes and add kb_group [step 1] #296
  • RAG: use scan instead of notify for file changes and add kb_group [step 2] #302
  • RAG: use scan instead of notify for file changes and add kb_group [step3] #304
  • Add server for Document #331
• 4. multi-embedding for one document
  • Support multi embedding for rag document #264
• 5. multi-documents for one Retriever
  • support multi-docs for Rertriever and support Document.create_node_group #256
• 6. add milvus store, add db index for db-store.
  • https://github.com/LazyAGI/LazyLLM/pulls/322
• 7. Support multi-index for one store
  • https://github.com/LazyAGI/LazyLLM/pulls/322
• 8. reranker function support float result (score)
• 9. Q-A pair search support
  • Add QA-DocNode #383
• 10. multi-media (picture first) support
  • rename DocNode::text to DocNode::content for image/video/... support #364
• 11. parallel transform
  • support multi-docs for Rertriever and support Document.create_node_group #256
• 12. create nodegroup with rule which specifies file name or file type
  • add pattern for create_node_group #263
• 13. add kb-group
  • RAG: use scan instead of notify for file changes and add kb_group [step 1] #296
  • RAG: use scan instead of notify for file changes and add kb_group [step 2] #302
  • RAG: use scan instead of notify for file changes and add kb_group [step3] #304
  • Add server for Document #331
• 14. embedding parallel
  • Support Infinity #300
• 15. support metadata filter (path / classfication / docid / ...)
  • add filter support for RAG #338)

[wzh1994]

can use https://github.com/fsspec/filesystem_spec to support 2