Skip to content

Commit

Permalink
switch to tools (#50)
Browse files Browse the repository at this point in the history
  • Loading branch information
ronakice authored Jun 12, 2020
1 parent f621265 commit 3364e2f
Show file tree
Hide file tree
Showing 3 changed files with 6 additions and 14 deletions.
6 changes: 3 additions & 3 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[submodule "eval"]
path = eval
url = https://github.com/castorini/anserini-eval.git
[submodule "tools"]
path = tools
url = https://github.com/castorini/anserini-tools.git
8 changes: 0 additions & 8 deletions docs/experiments-msmarco-document.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,14 +95,6 @@ It is worth noting again that you might need to modify the batch size to best fi

Upon completion, the re-ranked run file `runs/run.monot5.doc_fh.dev.tsv` will be available in the `runs` directory.

We can use the official MS MARCO evaluation script to verify the MRR@10:

```
python eval/msmarco_eval.py data/msmarco_doc_ans_small/fh/qrels.dev.small.tsv runs/run.monot5.doc_fh.dev.tsv
```

You should see the same result.

We can modify the argument for `--dataset` to `data/msmarco_doc_ans_small/sh` to re-rank the second half of the dataset, and don't forget to change output file name.

The results are as follows:
Expand Down
6 changes: 3 additions & 3 deletions docs/experiments-msmarco-passage.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ unzip data/msmarco_ans_small.zip -d data
As a sanity check, we can evaluate the first-stage retrieved documents using the official MS MARCO evaluation script.

```
python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv data/msmarco_ans_small/run.dev.small.tsv
python tools/eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv data/msmarco_ans_small/run.dev.small.tsv
```

The output should be:
Expand Down Expand Up @@ -105,7 +105,7 @@ The re-ranked run file `run.monobert.ans_small.dev.tsv` will also be available i
We can use the official MS MARCO evaluation script to verify the MRR@10:

```
python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monobert.ans_small.dev.tsv
python tools/eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monobert.ans_small.dev.tsv
```

You should see the same result. Great, let's move on to monoT5!
Expand Down Expand Up @@ -145,7 +145,7 @@ Upon completion, the re-ranked run file `run.monot5.ans_small.dev.tsv` will be a
We can use the official MS MARCO evaluation script to verify the MRR@10:

```
python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monot5.ans_small.dev.tsv
python tools/eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monot5.ans_small.dev.tsv
```

You should see the same result.
Expand Down

0 comments on commit 3364e2f

Please sign in to comment.