Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

A simple way to create and evaluate given a 'task' in the catalog and python data structure #1413

Merged
merged 29 commits into from
Dec 11, 2024
Merged

A simple way to create and evaluate given a 'task' in the catalog and python data structure #1413

merged 29 commits into from
Dec 11, 2024

Conversation

yoavkatz
Copy link
Member

@yoavkatz yoavkatz commented Dec 4, 2024

  • added api methods to create and evaluate datasets from python data structures
  • created simplfied qa task and qa metric catalogs
  • added default templates to Task, so user does need to specifiy it
  • added input checks in llm_as_judges and standard recipe

@yoavkatz
- added api methods to create and evaluate datasets from python data …
286c306 
…structures

- created simplfied qa task and qa metric catalogs
- added default templates to Task, so user does need to specifiy it
- added input checks in llm_as_judges and standard recipe

Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz yoavkatz marked this pull request as draft December 4, 2024 09:10
yoavkatz added 12 commits December 4, 2024 14:23
@yoavkatz
Added example of create_and_evaluate for rag
bd097c3 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Added type to Rag end to end response
992b731 
Also added documentation

Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Added option for integer ids in Retrieval metrics
b435695 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Moved to integer context_ids
4a077cf 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Improved Task field mismtach erorr message
13bdd25 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Added option to pass post processor to create_and_evaluate_dataset
5c213c0 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Addded descriptions and default values to classification
be43a13 
Added example of classification

Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Added descriptions
f804f1f 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Reorganized extracted qa task and templates.
ee706d3 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Updated json
93c7688 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Added documentation of default task and improved error message
6609dee 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Updated extraction tempalte
6a212f4 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz yoavkatz marked this pull request as ready for review December 11, 2024 08:56
yoavkatz and others added 13 commits December 11, 2024 10:56
@yoavkatz
Merge branch 'main' into simplified-eval-api
342645a
@yoavkatz
Upated recommended template
7e2ff02 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Sikplified and improved LLM as Judge
8d75f9f 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Improved error message.
76f4258 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Fixed classification default task
66cad6d 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Merge branch 'main' into simplified-eval-api
c6e8a06
@yoavkatz
Fixed classification default task
7742232 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Fixed unitxt
4de3fd7 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Fixed typo in template
cecca37 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Improved documentation
05d5ace 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Simplified API (removed create_and_evaluate_dataset)
9ac9089 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Improved error message
6327513 
Signed-off-by: Yoav Katz <[email protected]>
@yoavkatz
Changed wikitq back to extractive.
bb2217c 
Signed-off-by: Yoav Katz <[email protected]>
elronbandel
elronbandel reviewed Dec 11, 2024
View reviewed changes
docs/docs/adding_task.rst
input_fields={"num1" : int, "num2" : int},
reference_fields={"sum" : int},
prediction_type=int,
metrics=[
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe worth adding link to all available metrics

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand. We don't have this metric implemented.

@yoavkatz
Fixed examples
359224e 
Signed-off-by: Yoav Katz <[email protected]>
elronbandel
elronbandel requested changes Dec 11, 2024
View reviewed changes
Copy link
Member

@elronbandel elronbandel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mainly minor comments.

@yoavkatz yoavkatz enabled auto-merge (squash) December 11, 2024 15:28
@yoavkatz yoavkatz merged commit 3019204 into main Dec 11, 2024
17 checks passed
@yoavkatz yoavkatz deleted the simplified-eval-api branch December 11, 2024 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@elronbandel elronbandel elronbandel approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yoavkatz @elronbandel