You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently our model and dataset retrievers are not perfect, and it would be good to have a way to make them better.
One way we can do so is by explicitly training the model/dataset retrievers to:
Retrieve multiple datasets (models) and run the prompt2model pipeline with all of them
Take the resulting accuracy scores, and train the retriever so that the retriever gives higher scores to datasets (models) that give higher accuracy scores for the full pipeline
This would result in a training objective that explicitly rewards retrieving of datasets (models) that give high accuracy.
This would also be helpful for #285 , as it would reduce the need for human intervention when selecting models.
The text was updated successfully, but these errors were encountered:
Vijay and I actually thought about using LLM to automatically select columns and datasets, but just by prompting a row LLM, it is somehow impractical. Now, with DSPy, it seems that we can achieve this.
Currently our model and dataset retrievers are not perfect, and it would be good to have a way to make them better.
One way we can do so is by explicitly training the model/dataset retrievers to:
This would result in a training objective that explicitly rewards retrieving of datasets (models) that give high accuracy.
This would also be helpful for #285 , as it would reduce the need for human intervention when selecting models.
The text was updated successfully, but these errors were encountered: