Feature/msmarco psg

[crystina-z]

Just sending this PR to better track the progress :) Don't worry about it now

Now running ms marco psg while only reranking the top100 data looks right.

Confusing stuff to solve

• So far none of the runs on reranking top1k could save the checkpoint even tho it seems finished in some cases (but it can be saved in reranking the top100 case). While i was suspecting this is due to the running time limit in slurm, yet everytime queueing up a 4 days runs some other weird bug can pop up so after so many days it is still unconfirmed...
• The most recent run throws this error, while the sampler.generate_example and sampler.get_preds_in_trec_format seem to align with each other, and the dev records are prepared in this run so it's not because of overdue cache data. Still checking what's happening here. (Again this does not happen for the reranking top100 case.)

Traceback (most recent call last):
  File "run.py", line 108, in <module>
    task_entry_function()
  File "/home/czhang/aaai2021/tmp/capreolus/capreolus/task/rerank.py", line 54, in train
    return self.rerank_run(best_search_run, self.get_results_path())
  File "/home/czhang/aaai2021/tmp/capreolus/capreolus/task/rerank.py", line 101, in rerank_run
    self.benchmark.relevance_level,
  File "/home/czhang/aaai2021/tmp/capreolus/capreolus/trainer/tensorflow.py", line 205, in train
    trec_preds = self.get_preds_in_trec_format(dev_predictions, dev_data)
  File "/home/czhang/aaai2021/tmp/capreolus/capreolus/trainer/tensorflow.py", line 429, in get_preds_in_trec_format
    pred_dict[qid][docid] = predictions[i].numpy().astype(np.float16).item()
IndexError: list index out of range

Features/Support to add

• right now I have to hack the code to let the ranker only do evaluation on dev qids, otherwise the evaluation there (on millions of training queries!) gonna block the process forever. Probably this is already solved in feature/fit branch where only test qids are evaluated?
• use sqlite to prepare and store the docid2passage, so this can be run on the RAM-poor machines
• (maybe less urgent) handle the msmarco downloading using the allenai/ir_datasets

Sidenote (about the running time of some operation)

1. evaluation (on only dev set, not including the training queries)
   1.1 pytrec_eval: 40 sec
   1.2 (trec_eval: 4 secs)
2. Loading the whole runfile (including training data): ~310s (Saercher.load_trec_run())
3. Preparing the passage: ?
4. Prepare training runs: 3.5 hours
5. tfrecords (train + dev): 2 hours
6. training:
   6.1 (3k iteration) < 2 hours
   6.2 (30k iter) 11~12 hours
7. inference
   7.1 (top100): several hours
   7.2 (top1k) > 1.5 days

[lgtm-com]

This pull request introduces 3 alerts when merging 649804e into bf50423 - view on LGTM.com

new alerts:

• 3 for Unused import

[lgtm-com]

This pull request introduces 2 alerts when merging 7664df9 into 18e31b7 - view on LGTM.com

new alerts:

• 2 for Unused import

[lgtm-com]

This pull request introduces 1 alert when merging 569276e into 1767d5a - view on LGTM.com

new alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert when merging 1bbf0f2 into 1767d5a - view on LGTM.com

new alerts:

• 1 for Unused import

[andrewyates]

looks good! I left some minor questions/comments

[crystina-z]

actually do we just keep the headings as it is?

[lgtm-com]

This pull request introduces 1 alert when merging bbe134e into 1767d5a - view on LGTM.com

new alerts:

• 1 for Unused import