-
Notifications
You must be signed in to change notification settings - Fork 289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Polish tasks (PL-MTEB) #137
Conversation
This looks amazing, great job! 🚀
|
I searched but couldn't find any Polish Reranking tasks. Unfortunately, mMarco doesn't include Polish.
Of course, I will be grateful :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me! I've added the tab to the leaderboard: https://huggingface.co/spaces/mteb/leaderboard
Lmk if you'd change sth!
Two more things before merging
- Can you add each dataset to the README.md table? You can consult this script if you want to: https://github.com/embeddings-benchmark/mteb/blob/main/scripts/data/create_task_table.py
- Can you maybe run a few more models? I think running multilingual-e5-large would be great
- Can you run BEIR-PL Retrieval for the models too, so we have the average score? You will have to checkout to this branch
Once done, will merge this & BEIR-PL. Really amazing work!!!
Nice the code is good now - Merging this! Let me know if you manage to run the additional models/evals! |
This includes points for 12 datasets (24) across 4 tasks (8). These points are given to rafalposwiata and then one point for review
* docs: Added missing points for #214 Added 6x2 points for guenthermi for datasets and 1 point to Muennighoff for review I have not accounted for bonus points as I am not sure was what available at the time. * docs: added point for #197 Added 2 points for rasdani and 2 bonus points for the first german retrieval (I believe). Added one point for each of the reviewers * docs: added points for #116 This includes 6 points for 3 datasets to slvnwhrl +2 for first german clustering task also added points for reviews * Added points for #134 cmteb This includes 29 datasets (38 points) and 6x2 bonus points (12 points) for the 6 taskXlanguage which was not previously included. All the points are attributed to @staoxiao, though we can split them if needed. We also added points for review. * docs: Added points for #137 polish This includes points for 12 datasets (24) across 4 tasks (8). These points are given to rafalposwiata and then one point for review * docs: Added points for #27 (spanish) These include 9 datasets (18 points) across 4 news tasks (8) for spanish. Points are given to violenil as the contributor, and one points for reviewers. Points can be split up if needed. * docs: Added points for #224 Added points 2 points for the dataset. I could imagine that I might have missed some bonus points as well. Also added one point for review. * docs: Added points for #210 (korean) This include 3 datasets (6 points) across 1 new task (+2 bonus) for korean. Also added 1 points for reviewers. * Add contributor --------- Co-authored-by: Niklas Muennighoff <[email protected]>
Add Polish tasks (PL-MTEB).
Files structure was inspired by Chinese C-MTEB.
Evaluation results for the three models have already been added to https://huggingface.co/datasets/mteb/results