Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added MMTEB #275

Merged
merged 14 commits into from
Mar 24, 2024
Merged

Added MMTEB #275

merged 14 commits into from
Mar 24, 2024

Conversation

KennethEnevoldsen
Copy link
Contributor

@KennethEnevoldsen KennethEnevoldsen commented Mar 21, 2024

We are getting ready to add MMTEB, so I have added it here to make sure that everything links up. I still consider it a work in progress so it is not linked it in the readme.

This PR:

  • Add the mmteb folder
  • Add documentation for adding datasets
    • Updates existing documentation
    • Added docstring to metadata object
  • Moved metadata_dict attribute to abstask
  • Added the point system. Did a slight refactor here, let me know what you think
  • Added minor tweaks to the MMTEB description (mostly fixing links)

Copy link
Contributor

@Muennighoff Muennighoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is amazing! 🚀🙌

docs/adding_a_task.md Outdated Show resolved Hide resolved
docs/adding_a_task.md Outdated Show resolved Hide resolved
docs/adding_a_task.md Outdated Show resolved Hide resolved
docs/adding_a_task.md Outdated Show resolved Hide resolved
docs/adding_a_task.md Outdated Show resolved Hide resolved
docs/mmteb/points.md Outdated Show resolved Hide resolved
docs/mmteb/points.md Outdated Show resolved Hide resolved
README.md Show resolved Hide resolved
docs/mmteb/points.md Outdated Show resolved Hide resolved
Copy link
Contributor

@MartinBernstorff MartinBernstorff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work, Kenneth! Especially moving the property to a parent class deduplicated a lot of code 👍

docs/adding_a_dataset.md Outdated Show resolved Hide resolved
docs/adding_a_dataset.md Outdated Show resolved Hide resolved
docs/mmteb/readme.md Outdated Show resolved Hide resolved
docs/mmteb/readme.md Outdated Show resolved Hide resolved
docs/mmteb/readme.md Outdated Show resolved Hide resolved
docs/mmteb/readme.md Outdated Show resolved Hide resolved
docs/mmteb/readme.md Outdated Show resolved Hide resolved
mteb/abstasks/TaskMetadata.py Show resolved Hide resolved
mteb/abstasks/TaskMetadata.py Show resolved Hide resolved
mteb/abstasks/TaskMetadata.py Show resolved Hide resolved
@KennethEnevoldsen
Copy link
Contributor Author

I have updated the PR. Just swinging it out for a final review, before merging it in (the discussion can still continue)

Copy link
Member

@loicmagne loicmagne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice project!

docs/mmteb/readme.md Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
.github/pull_request_template.md Outdated Show resolved Hide resolved
README.md Outdated Show resolved Hide resolved
.github/pull_request_template.md Outdated Show resolved Hide resolved
Comment on lines +1 to +18
# Points

| GitHub | Total points | New dataset | New task | Dataset annotations | (Bug)fixes | Running Models | Review PR | Paper Writing | Ideation | Coordination |
| ----------------- | ------------ | ----------- | -------- | ------------------- | ---------- | -------------- | -------- | -------------- | -------- | ------------- |
| KennethEnevoldsen | | 38 | | | | | | | | |
| x-tabdeveloping | | 2 | | | | | | | | |

Note that coordination and ideation is not included in the total points, but is used to determine first and last authors.

# Contributor Informations

First name and last name are as you want them to appear in a publication.

| GitHub | First name | Last name | Email | User on openreview |
| ----------------- | ---------- | ---------- | ---------------------------- | -------------------- |
| KennethEnevoldsen | Kenneth | Enevoldsen | [email protected] | ~Kenneth_Enevoldsen1 |
| x-tabdeveloping | Márton | Kardos | | ~Márton_Kardos1 |

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could someone help retroactively add everyone who has contributed in the past here? cc @imenelydiaker @wissam-sib

docs/mmteb/readme.md Show resolved Hide resolved
@KennethEnevoldsen
Copy link
Contributor Author

Hei everyone, will merge this in (as it is e.g. holding up other PRs), but do feel free to open new PRs with corrections

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants