More robust Reader eval by limiting max answers and creating no answer labels

[brandenchan]

When evaluating using Reader.eval() we read data from file, store it in an ElasticsearchDocumentStore, reconstruct input dictionaries and then evaluate using a farm.Evaluator object.

Two problems were arising, linked to the DocumentStore.

1. When a duplicate question is asked on the same document (see squad dev set "What is the CJEU's duty?"), indexing into and fetching from the DocumentStore causes their labels to be merged. This can result in a list of answers longer than the default max number of answers 6. A quick solution is implemented here to reduce the list down to 6.

2. Only span labels are indexed into the DocumentStore and fetched, meaning that we lose the no_answer samples. This PR ensures that question which don't have a corresponding label in the DocumentStore are assigned a "no_answer" label.

Neither of these problems affect Reader.eval_on_file()

This PR also resolves a performance discrepancy between Reader.eval(), Reader.eval_on_file() and farm.Evaluator. All now show the following performance:

 model = deepset/roberta-base-squad2
 data = nq_dev_subset_v2.json

 Reader Top-N-Accuracy: 0.7222222222222222
 Reader Exact Match: 0.4074074074074074
 Reader F1-Score: 0.4463589193057792

[brandenchan]

The latest commit solves the discrepancies in performance between Reader.eval() with no_answers and Reader.eval_on_file(). Previously, there were labels that were being lost during the process of indexing and retrieving labels to and from elasticsearch

[brandenchan]

Evaluating in FARM has a default QuestionAnsweringPredictionHead.n_best = 5.

Haystack's Reader.top_k_per_candidate = 3 by default and consequently, QuestionAnsweringPredictionHead.n_best = 4 by default.

The discrepancy between these two repos is resolved by setting the Reader.top_k_per_candidate = 4

[tholor]

Only tiny comment on the warning message. Rest looks great!

[tholor]

• What is the impact of this? As a user, it is unclear to me why this might be a problem. Is it only the top_k_accuracy metric that will be different to eval in FARM? If yes, I would rather make it an info than a warning.