Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Allow aggregated tasks within benchmarks #1771

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: Allow aggregated tasks within benchmarks #1771

wants to merge 1 commit into from

Conversation

KennethEnevoldsen
Copy link
Contributor

Fixes #1231

  •  added AbsTaskAggregated
  •  added CQADupstackRetrieval
  • updated mteb(eng, classic) to use CQADupstackRetrieval instead of its subtasks
  • refactor

Not at all sure this is the best way to implement it. My hunch is that there is a smarter way to list is such that the complexity is handled by MTEB.run. Will try to do this in the refactor.

Checklist

  • Run tests locally to make sure nothing is broken using make test.
  • Run the formatter to format the code using make lint.

Samoed
Samoed reviewed Jan 11, 2025
View reviewed changes
mteb/abstasks/aggregated_task.py
torch.manual_seed(self.seed)
torch.cuda.manual_seed_all(self.seed)

def evaluate(
Copy link
Collaborator

@Samoed Samoed Jan 11, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just run in loop for each task it's evaluate like load_data?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Samoed Samoed Samoed left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Allow aggregated tasks within benchmarks
2 participants
@KennethEnevoldsen @Samoed