RRF between RM3 and Ax

[lintool]

As a simple experiment, let's trying running RRF between RM3 and Ax: this package seems to already have RRF implemented: https://github.com/joaopalotti/trec_tools

@emmileaf this might not make much sense right now, but I'll walk you through this...

[lintool]

@emmileaf you showed me some results on this? Can you post to this thread?

[emmileaf]

(Results updated as of commit 3545350)

Reciprocal Rank Fusion

Robust04:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RRF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2895	0.2903	0.2963	0.019821	0.024997
P30	0.3333	0.3365	0.3410	0.037711	0.161866

Core17:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RRF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2787	0.2823	0.2921	0.019958	0.085324
P30	0.4980	0.5093	0.5087	0.244491	0.934356

Core18:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RRF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2920	0.3136	0.3118	0.039248	0.845065
P30	0.4027	0.4200	0.4227	0.117934	0.869243

[lintool]

@emmileaf Thanks! Can you try the other fusion methods offered by the package?

[emmileaf]

For sure- I'll add to this thread.

[emmileaf]

Other fusion methods from trectools:

(Results updated as of commit 3545350)

Rank Biased Precision

Robust04:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RBP)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2895	0.2903	0.2682	0.000000	0.000000
P30	0.3333	0.3365	0.3396	0.061813	0.345704

Core17:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RBP)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2787	0.2823	0.2436	0.000469	0.000372
P30	0.4980	0.5093	0.5140	0.131967	0.447918

Core18:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (RBP)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2920	0.3136	0.2764	0.189879	0.000268
P30	0.4027	0.4200	0.4240	0.110350	0.769898

Vector Space Fusion

Robust04:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (VSF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2895	0.2903	0.2901	0.073643	0.966559
P30	0.3333	0.3365	0.3336	0.670020	0.604390

Core17:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (VSF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2787	0.2823	0.2795	0.015275	0.792411
P30	0.4980	0.5093	0.4993	0.321492	0.495218

Core18:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (VSF)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2920	0.3136	0.2949	0.199128	0.2468215
P30	0.4027	0.4200	0.4060	0.440635	0.549512

Comb (Sum)

Robust04:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (SUM)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2895	0.2903	0.2925	0.000281	0.632728
P30	0.3333	0.3365	0.3352	0.208824	0.794289

Core17:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (SUM)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2787	0.2823	0.2829	0.021926	0.941396
P30	0.4980	0.5093	0.5027	0.145938	0.638465

Core18:

	BM25+AX	BM25+RM3	BM25+AX+RM3 (SUM)	P-value (Base: BM25+AX)	P-value (Base: BM25+RM3)
MAP	0.2920	0.3136	0.2971	0.067306	0.287336
P30	0.4027	0.4200	0.4060	0.608836	0.522277

[lintool]

so tl;dr - is that RRF still works the best?

[emmileaf]

Yup looks like it, at least for default params/etc. from the package.

[lintool]

Merging of #658 unblocked Core18
Per #668, Core17 probably needs to be rerun

[emmileaf]

Merging of #658 unblocked Core18
Per #668, Core17 probably needs to be rerun

• Results re-run and edited in comments above.
• TLDR from updated results: RRF still the most effective, but does not perform as well on core17/core18 compared to robust04.

[lintool]

Good to have these results, but moving forward this should probably be done on the Pyserini end...