Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Evaluation: Remove parallel from composite evaluators #38168

Merged
merged 12 commits into from
Oct 30, 2024
1 change: 1 addition & 0 deletions sdk/evaluation/azure-ai-evaluation/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
### Features Added

### Breaking Changes
- The `parallel` parameter has been removed from composite evaluators: `QAEvaluator`, `ContentSafetyChatEvaluator`, and `ContentSafetyMultimodalEvaluator`. To control evaluator parallelism, you can now use the `_parallel` keyword argument, though please note that this private parameter may change in the future.

### Bugs Fixed

Expand Down
2 changes: 1 addition & 1 deletion sdk/evaluation/azure-ai-evaluation/assets.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,5 +2,5 @@
"AssetsRepo": "Azure/azure-sdk-assets",
"AssetsRepoPrefixPath": "python",
"TagPrefix": "python/evaluation/azure-ai-evaluation",
"Tag": "python/evaluation/azure-ai-evaluation_e3ec13551e"
"Tag": "python/evaluation/azure-ai-evaluation_daf1ed16fc"
}
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,8 @@ def wrapper(*args: P.args, **kwargs: P.kwargs) -> EvaluationResult:
user_agent=USER_AGENT,
)

track_in_cloud = bool(pf_client._config.get_trace_destination()) # pylint: disable=protected-access
trace_destination = pf_client._config.get_trace_destination() # pylint: disable=protected-access
track_in_cloud = bool(trace_destination) if trace_destination != "none" else False
evaluate_target = bool(kwargs.get("target", None))
evaluator_config = bool(kwargs.get("evaluator_config", None))
custom_dimensions: Dict[str, Union[str, bool]] = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ class ContentSafetyEvaluator(EvaluatorBase):
# TODO address 3579092 to re-enabled parallel evals.
def __init__(self, credential, azure_ai_project, eval_last_turn: bool = False, **kwargs):
super().__init__(eval_last_turn=eval_last_turn)
self._parallel = kwargs.pop("parallel", False)
self._parallel = kwargs.pop("_parallel", False)
self._evaluators: List[Callable[..., Dict[str, Union[str, float]]]] = [
ViolenceEvaluator(credential, azure_ai_project),
SexualEvaluator(credential, azure_ai_project),
Expand Down

This file was deleted.

Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,8 @@ class ContentSafetyMultimodalEvaluator:
:param azure_ai_project: The scope of the Azure AI project, containing the subscription ID,
resource group, and project name.
:type azure_ai_project: ~azure.ai.evaluation.AzureAIProject
:param parallel: Specifies whether to use parallel execution for evaluators.
If True, evaluators execute in parallel; otherwise, they execute sequentially. Defaults to True.
:type parallel: bool
:param kwargs: Additional arguments to pass to the evaluator.
:type kwargs: Any

:return: A function that evaluates multimodal chat messages and generates content safety metrics.
:rtype: Callable
Expand Down Expand Up @@ -92,8 +91,8 @@ class ContentSafetyMultimodalEvaluator:

"""

def __init__(self, credential, azure_ai_project, parallel: bool = False):
self._parallel = parallel
def __init__(self, credential, azure_ai_project, **kwargs):
self._parallel = kwargs.pop("_parallel", False)
self._evaluators: List[Callable[..., Dict[str, Union[str, float]]]] = [
ViolenceMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
SexualMultimodalEvaluator(credential=credential, azure_ai_project=azure_ai_project),
Expand Down
Loading
Loading