Refactor MS MARCO (V1) doc regressions (#1723)

Rebuild all doc regressions from scratch.

README.md

@@ -54,16 +54,16 @@ For the most part, these runs are based on [_default_ parameter settings](https:
   + Bag-of-words models: [baselines](docs/regressions-msmarco-passage.md), [doc2query](docs/regressions-msmarco-passage-doc2query.md), [doc2query-T5](docs/regressions-msmarco-passage-docTTTTTquery.md)
   + Sparse learned models: [DeepImpact](docs/regressions-msmarco-passage-deepimpact.md), [uniCOIL with doc2query-T5](docs/regressions-msmarco-passage-unicoil.md), [uniCOIL with TILDE](docs/regressions-msmarco-passage-unicoil-tilde-expansion.md), [SPLADEv2](docs/regressions-msmarco-passage-distill-splade-max.md)
 + Regressions for MS MARCO (V1) Document Ranking:
-  + Per doc method: [baselines](docs/regressions-msmarco-doc.md), [doc2query-T5](docs/regressions-msmarco-doc-docTTTTTquery-per-doc.md)
-  + Per passage method: [baselines](docs/regressions-msmarco-doc-per-passage.md) ([v2](docs/regressions-msmarco-doc-per-passage-v2.md), [v3](docs/regressions-msmarco-doc-per-passage-v3.md))[*](docs/experiments-msmarco-doc-doc2query-details.md), [doc2query-T5](docs/regressions-msmarco-doc-docTTTTTquery-per-passage.md) ([v3](docs/regressions-msmarco-doc-docTTTTTquery-per-passage-v3.md))[*](docs/experiments-msmarco-doc-doc2query-details.md)
+  + Complete doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-msmarco-doc.md), [doc2query-T5](docs/regressions-msmarco-doc-docTTTTTquery.md)
+  + Segmented doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-msmarco-doc-segmented.md), [doc2query-T5](docs/regressions-msmarco-doc-segmented-docTTTTTquery.md)
 + Regressions for TREC 2019 Deep Learning Track:
   + Passage ranking: [baselines](docs/regressions-dl19-passage.md), [doc2query-T5](docs/regressions-dl19-passage-docTTTTTquery.md)
-  + Document ranking, per doc method: [baselines](docs/regressions-dl19-doc.md), [doc2query-T5](docs/regressions-dl19-doc-docTTTTTquery-per-doc.md)
-  + Document ranking, per passage method: [baselines](docs/regressions-dl19-doc-per-passage.md), [doc2query-T5](docs/regressions-dl19-doc-docTTTTTquery-per-passage.md)
+  + Document ranking, complete doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl19-doc.md), [doc2query-T5](docs/regressions-dl19-doc-docTTTTTquery.md)
+  + Document ranking, segmented doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl19-doc-segmented.md), [doc2query-T5](docs/regressions-dl19-doc-segmented-docTTTTTquery.md)
 + Regressions for TREC 2020 Deep Learning Track:
   + Passage ranking: [baselines](docs/regressions-dl20-passage.md), [doc2query-T5](docs/regressions-dl20-passage-docTTTTTquery.md)
-  + Document ranking, per doc method: [baselines](docs/regressions-dl20-doc.md), [doc2query-T5](docs/regressions-dl20-doc-docTTTTTquery-per-doc.md)
-  + Document ranking, per passage method: [baselines](docs/regressions-dl20-doc-per-passage.md), [doc2query-T5](docs/regressions-dl20-doc-docTTTTTquery-per-passage.md)
+  + Document ranking, complete doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl20-doc.md), [doc2query-T5](docs/regressions-dl20-doc-docTTTTTquery.md)
+  + Document ranking, segmented doc[*](docs/experiments-msmarco-doc-doc2query-details.md): [baselines](docs/regressions-dl20-doc-segmented.md), [doc2query-T5](docs/regressions-dl20-doc-segmented-docTTTTTquery.md)
 + Regressions for MS MARCO (V2) Passage Ranking:
   + Bag-of-words models: [baselines](docs/regressions-msmarco-v2-passage.md), [on augmented corpus](docs/regressions-msmarco-v2-passage-augmented.md)
   + Sparse learned models: [uniCOIL noexp zero-shot](docs/regressions-msmarco-v2-passage-unicoil-noexp-0shot.md)

docs/experiments-msmarco-doc-doc2query-details.md

@@ -1,4 +1,4 @@
-# Anserini: Reproducibility Notes for MS MARCO V1 Doc Ranking
+# Anserini: Reproducibility Notes for MS MARCO V1
 
 <blockquote class="twitter-tweet"><p lang="en" dir="ltr">Reproducibility is hard.</p>&mdash; Jimmy Lin (@lintool) <a href="https://twitter.com/lintool/status/1458853999298465796?ref_src=twsrc%5Etfw">November 11, 2021</a></blockquote>
 
@@ -22,8 +22,22 @@ This was for dense retrieval experiments, as we were not aware of the doc2query-
 It is very likely, but we cannot know for sure, that this was the same segmentation that generated the original doc2query-T5 expansions.
 Fortunately, Xueguang was able to save a copy of this segmented corpus.
 
-So, now we have:
+---
 
-+ `doc-per-passage-v2`: materialized corpus with 20,545,677 segments.
-+ `doc-per-passage-v3`: same as above, except with URL. Note that bag-of-words search over this variant yields higher effectiveness than above, but for input to an encoder, you probably don't want to include the URL.
-+ `doc-docTTTTTquery-per-doc-v3`: `doc-per-passage-v3`, but with the doc2query-T5 expansions added in. 
+In January 2022, we completely refactored the doc2query-T5 expansion data for the MS MARCO (V1) corpora.
+They are now available as Huggingface Datasets:
+
++ [`msmarco_v1_passage_doc2query-t5_expansions`](https://huggingface.co/datasets/castorini/msmarco_v1_passage_doc2query-t5_expansions): passage expansions
++ [`msmarco_v1_doc_doc2query-t5_expansions`](https://huggingface.co/datasets/castorini/msmarco_v1_doc_doc2query-t5_expansions): document expansions
++ [`msmarco_v1_doc_segmented_doc2query-t5_expansions`](https://huggingface.co/datasets/castorini/msmarco_v1_doc_segmented_doc2query-t5_expansions): document segment expansions
+
+So now we have the following new regressions:
+
++ `msmarco-doc`: document corpus in Anserini's jsonl format with 3,213,835 documents. Each contains URL, title, body, delimited by newlines.
++ `msmarco-doc-docTTTTTquery`: same as above, but with docTTTTTquery expansions, delimited by another newline.
++ `msmarco-segmented`: segmented document corpus in Anserini's jsonl format with 20,545,677 segments. Each contains URL, title, segment, delimited by newlines.
++ `msmarco-segmented-docTTTTTquery`: same as above, but with docTTTTTquery expansions, delimited by another newline.
+
+These new versions yield end-to-end scores that are slightly different, so if numbers reported in a paper do not exactly match, this may be the reason.
+
+*TODO:* Circle back and add links to scripts once everything has been verified and checked in.

docs/regressions-backgroundlinking18.md

@@ -12,7 +12,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection WashingtonPostCollection \
   -input /path/to/wapo.v2 \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -generator WashingtonPostGenerator \
   -threads 1 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.wapo.v2 &
@@ -34,19 +34,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking18.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25.topics.backgroundlinking18.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking18.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25+rm3.topics.backgroundlinking18.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking18.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25+rm3+df.topics.backgroundlinking18.txt \
   -backgroundlinking -backgroundlinking.datefilter -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &

docs/regressions-backgroundlinking19.md

@@ -12,7 +12,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection WashingtonPostCollection \
   -input /path/to/wapo.v2 \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -generator WashingtonPostGenerator \
   -threads 1 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.wapo.v2 &
@@ -34,19 +34,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking19.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25.topics.backgroundlinking19.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking19.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25+rm3.topics.backgroundlinking19.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v2 \
+  -index indexes/lucene-index.wapo.v2/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking19.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v2.bm25+rm3+df.topics.backgroundlinking19.txt \
   -backgroundlinking -backgroundlinking.datefilter -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &

docs/regressions-backgroundlinking20.md

@@ -12,7 +12,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection WashingtonPostCollection \
   -input /path/to/wapo.v3 \
-  -index indexes/lucene-index.wapo.v3 \
+  -index indexes/lucene-index.wapo.v3/ \
   -generator WashingtonPostGenerator \
   -threads 1 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.wapo.v3 &
@@ -34,19 +34,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v3 \
+  -index indexes/lucene-index.wapo.v3/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking20.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v3.bm25.topics.backgroundlinking20.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v3 \
+  -index indexes/lucene-index.wapo.v3/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking20.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v3.bm25+rm3.topics.backgroundlinking20.txt \
   -backgroundlinking -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.wapo.v3 \
+  -index indexes/lucene-index.wapo.v3/ \
   -topics src/main/resources/topics-and-qrels/topics.backgroundlinking20.txt -topicreader BackgroundLinking \
   -output runs/run.wapo.v3.bm25+rm3+df.topics.backgroundlinking20.txt \
   -backgroundlinking -backgroundlinking.datefilter -backgroundlinking.k 100 -bm25 -rm3 -hits 100 &

docs/regressions-car17v1.5.md

@@ -12,7 +12,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection CarCollection \
   -input /path/to/car-paragraphCorpus.v1.5 \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -generator DefaultLuceneDocumentGenerator \
   -threads 1 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.car-paragraphCorpus.v1.5 &
@@ -35,37 +35,37 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.bm25.topics.car17v1.5.benchmarkY1test.txt \
   -bm25 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.bm25+rm3.topics.car17v1.5.benchmarkY1test.txt \
   -bm25 -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.bm25+ax.topics.car17v1.5.benchmarkY1test.txt \
   -bm25 -axiom -axiom.deterministic -rerankCutoff 20 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.ql.topics.car17v1.5.benchmarkY1test.txt \
   -qld &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.ql+rm3.topics.car17v1.5.benchmarkY1test.txt \
   -qld -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v1.5 \
+  -index indexes/lucene-index.car-paragraphCorpus.v1.5/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v1.5.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v1.5.ql+ax.topics.car17v1.5.benchmarkY1test.txt \
   -qld -axiom -axiom.deterministic -rerankCutoff 20 &

docs/regressions-car17v2.0-doc2query.md

@@ -18,7 +18,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection JsonCollection \
   -input /path/to/car-paragraphCorpus.v2.0-doc2query \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -generator DefaultLuceneDocumentGenerator \
   -threads 30 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.car-paragraphCorpus.v2.0-doc2query &
@@ -41,37 +41,37 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.bm25.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.bm25+rm3.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.bm25+ax.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 -axiom -axiom.deterministic -rerankCutoff 20 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.ql.topics.car17v2.0.benchmarkY1test.txt \
   -qld &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.ql+rm3.topics.car17v2.0.benchmarkY1test.txt \
   -qld -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0-doc2query/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0-doc2query.ql+ax.topics.car17v2.0.benchmarkY1test.txt \
   -qld -axiom -axiom.deterministic -rerankCutoff 20 &

docs/regressions-car17v2.0.md

@@ -12,7 +12,7 @@ Typical indexing command:
 target/appassembler/bin/IndexCollection \
   -collection CarCollection \
   -input /path/to/car-paragraphCorpus.v2.0 \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -generator DefaultLuceneDocumentGenerator \
   -threads 1 -storePositions -storeDocvectors -storeRaw \
   >& logs/log.car-paragraphCorpus.v2.0 &
@@ -35,37 +35,37 @@ After indexing has completed, you should be able to perform retrieval as follows
 
 ```
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.bm25.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.bm25+rm3.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.bm25+ax.topics.car17v2.0.benchmarkY1test.txt \
   -bm25 -axiom -axiom.deterministic -rerankCutoff 20 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.ql.topics.car17v2.0.benchmarkY1test.txt \
   -qld &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.ql+rm3.topics.car17v2.0.benchmarkY1test.txt \
   -qld -rm3 &
 
 target/appassembler/bin/SearchCollection \
-  -index indexes/lucene-index.car-paragraphCorpus.v2.0 \
+  -index indexes/lucene-index.car-paragraphCorpus.v2.0/ \
   -topics src/main/resources/topics-and-qrels/topics.car17v2.0.benchmarkY1test.txt -topicreader Car \
   -output runs/run.car-paragraphCorpus.v2.0.ql+ax.topics.car17v2.0.benchmarkY1test.txt \
   -qld -axiom -axiom.deterministic -rerankCutoff 20 &