How to evaluate retriever?

[tholor]

so I have another question,I have context, Question, answer triples same as SQUAD dataset in non-english language. am done training it for reader module, so given the retriver here how can we integrat it, since Documents and Passages of retriver are coming from different data, then reader is trained ON, so how can we have evaluate performance of retriver module?

(originally posted by @samreenkazi in #334)

┆Issue is synchronized with this Jira Task by Unito

[tholor]

so how can we have evaluate performance of retriver module?

If you have your custom dataset in SQuAD format, you can evaluate the retriever as shown in Tutorial 5.
You basically add the eval data to your DocumentStore and then run eval:

document_store.add_eval_data("../data/nq/nq_dev_subset_v2.json")
...
retriever.eval()

See the tutorial for details.

(Just be aware that the metrics of reader.eval() are currently broken and show very low numbers. We work on a fix in #331. The retriever metrics that you are interested in are correct)

Did that answer your question @samreenkazi?

[samreenkazi]

@tholor Question is I have( Q,P,A )triple as the training dataset, the reader is trained on it and it was created from Wikipedia dump three years ago, In what form should we give query to retriever and how do we evaluate the result since we don't have ground truth passage in that case

[tholor]

we don't have ground truth passage

Not sure if I fully understand. When you have triples of Query, Passage, Answer you do have some ground truth data. You can basically take one triple, give the Query to the Retriever, and then check whether the Passage is within the top-k results of the retriever. Or am I missing anything here? If that's what you are after, see my post above for hints how to do that in haystack.

[tholor]

Closing due to inactivity. Feel free to re-open.