Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Evaluation export of query pipeline so we can integrate with Ragas #232

Closed
timonv opened this issue Aug 13, 2024 · 0 comments
Closed

Evaluation export of query pipeline so we can integrate with Ragas #232

timonv opened this issue Aug 13, 2024 · 0 comments
Assignees
@timonv
Labels
enhancement New feature or request

Comments

@timonv
Copy link
Member

timonv commented Aug 13, 2024

Is your feature request related to a problem? Please describe.
Evaluation is still lacking in swiftide and Rust is not that great at doing data analytics.

Describe the solution you'd like
Integrate with Ragas by exporting the query dataframe as per their documentation from the query pipeline to csv.

  • Query pipelines can have an evaluator
  • Ragas would be the first one
  • Evaluators are responsible for the whole setup

Additional context
Issue over at Ragas explodinggradients/ragas#1165
Dataframe format https://docs.ragas.io/en/v0.0.22/howtos/applications/data_preparation.html
@timonv timonv mentioned this issue Aug 14, 2024
Merged
@timonv timonv self-assigned this Aug 15, 2024
@timonv timonv added the enhancement New feature or request label Aug 16, 2024
timonv added a commit that referenced this issue Aug 29, 2024
@timonv
feat(query): Ragas support (#236)
c00b6c8 
Work in progress on support for ragas as per
explodinggradients/ragas#1165 and #232

Add an optional evaluator to a pipeline. Evaluators need to handle
transformation events in the query pipeline. The Ragas evaluator
captures the transformations as per
https://docs.ragas.io/en/latest/howtos/applications/data_preparation.html.

You can find a working notebook here
https://github.com/bosun-ai/swiftide-tutorial/blob/c510788a625215f46575415161659edf26fc1fd5/ragas/notebook.ipynb
with a pipeline using it here
bosun-ai/swiftide-tutorial#1



TODO:
- [x] Test it with Ragas
- [x] Add more tests
@timonv timonv closed this as completed Sep 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@timonv timonv

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@timonv