Multimodal Representation Benchmark

Installation

pip install .

Usage

python -m src.main

Support Tasks

• Retrieval
  • M-BEIR
  • 🚧 AToMiC (Under Construction) 🚧
• 🚧 Classification (Under Construction) 🚧

Acknowledgement

We would like to acknowledge the following resources that greatly contributed to our project:

• UniIR: The primary dataset our work is built upon.
• MTEB: We have extended its excellent work and ideas to a broader multimodal representation evaluation.
• ranx: The blazing fast IR evaluation toolkit.
• transformers, datasets: This project would not be possible without the great work and resource from the Hugging Face teams.
• And of course, all the unknown heroes of the foundational libraries