Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add batch evaluation method for pipelines #2942
feat: add batch evaluation method for pipelines #2942
Changes from 21 commits
9d1f2dd
4a398a3
59b64c3
789cc2d
9a6c681
671826a
ad40a55
1bda44b
7bacd93
8cb7a51
5cf8f0a
b3b57f5
9aad547
313a5e8
2738994
d184d20
07682eb
a2d4d6f
195f8a1
17f750d
16076ea
7339b45
0fa1bcb
a1ac6b4
afd03a5
4b0b242
ab4dccb
af434ee
16cf698
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
Large diffs are not rendered by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need this parameter
add_isolated_node_eval
? As a user of this API it wasn't clear to me immediately what it is about and why do we need it?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, we need it. It's the same parameter as in the standard
run()
. If it is set toTrue
, the evaluation is executed with labels as node inputs in addition to the integrated evaluation, where the node inputs are the outputs of the previous node in the pipeline.Large diffs are not rendered by default.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I traced the
sas_use_gpu
parameter being passed to CrossEncoder viasemantic_answer_similarity
method. Let's keep in mind that we should soon replace alluse_gpu
parameter withdevices
parameter (as per #3062 and #2826) Just to keep in mind as todo item.