Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RAG: Get relevance score when using Reranker Module #381

Open
ChenJiahaoST opened this issue Dec 5, 2024 · 1 comment
Open

RAG: Get relevance score when using Reranker Module #381

ChenJiahaoST opened this issue Dec 5, 2024 · 1 comment

Comments

@ChenJiahaoST
Copy link

Hi, I just found that I can't get relevance scores when using reranker(ModuleReranker), I can just get a sorted_indices like "[0, 3, 1, 2]" and get topk DocNodes

reranker_model = Reranker('ModuleReranker', model=bge-reranker-large
reranker_model.start()
nodes = [
        DocNode(text="深度学习是..."),
        DocNode(text="机器学习是..."),
        DocNode(text="自然语言处理是..."),
        DocNode(text="计算机视觉是..."),
    ]
query = "深度学习是什么?"
reranker_model(nodes, query)

But when I use other reranker model , I can get a series of relevance socre telling the relevance between node and query so that I can set a threshold to filter some bad nodes to optimize the generation performance of my RAG system.

like (https://jina.ai/reranker/):

{
  "model": "jina-reranker-v2-base-multilingual",
  "usage": {
    "total_tokens": 48
  },
  "results": [
    {
      "index": 0,
      "document": {
        "text": "深度学习是..."
      },
      "relevance_score": 0.6261242032051086
    },
    {
      "index": 2,
      "document": {
        "text": "自然语言处理是..."
      },
      "relevance_score": 0.1037486344575882
    },
    {
      "index": 1,
      "document": {
        "text": "机器学习是..."
      },
      "relevance_score": 0.09009299427270889
    },
    {
      "index": 3,
      "document": {
        "text": "计算机视觉是..."
      },
      "relevance_score": 0.08035746961832047
    }
  ]
}

Can I get the relevance score like this by using LazyLLM's Rerankers? Not just topk

@wzh1994
Copy link
Contributor

wzh1994 commented Dec 5, 2024

这个功能在计划中,issue见#155 的第8条,我们会尽快支持

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants