You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
When running pipeline.eval(), we have have two options to get metrics from our nodes: isolated and integrated.
While integrated works already on our generative QA nodes, the isolated mode is currently only working for Readers not Generators.
Example:
When running pipeline.eval() on a pipeline consisting of a BM25Retriever -> OpenAIAnswerGenerator, we get the error message No node(s) or global parameter(s) named add_isolated_node_eval found in pipeline.
Describe the solution you'd like
Modify the run() method of the base class to use the perfect labels as input from previous nodes
Additional context
This will also be needed to allow experiment runs in deepset Cloud as the additional isolated mode there is the expected default.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
When running pipeline.eval(), we have have two options to get metrics from our nodes: isolated and integrated.
While integrated works already on our generative QA nodes, the isolated mode is currently only working for Readers not Generators.
Example:
When running
pipeline.eval()
on a pipeline consisting of aBM25Retriever
->OpenAIAnswerGenerator
, we get the error messageNo node(s) or global parameter(s) named add_isolated_node_eval found in pipeline.
Describe the solution you'd like
Modify the run() method of the base class to use the perfect labels as input from previous nodes
Additional context
This will also be needed to allow experiment runs in deepset Cloud as the additional isolated mode there is the expected default.
The text was updated successfully, but these errors were encountered: