Leverage async batch run for first async-enabled evaluator - FluencyEvaluator

[ninghu]

Description

This PR aims to improve the performance of the evaluate API by leveraging async batch run to eliminate the overhead associated with using multiple processes. The key changes include:

• Converting the FluencyEvaluator to an async-based implementation.
• Plumbing work in the BatchEngine to enable async batch runs.

For more details, please check the "Run Evaluators Asynchronously" section in this document.

Results:

• Evaluation with 1 evaluator and 1 row used to take 16 seconds. Now, it takes only 2 seconds, about 87% improve.
• The result is very close to a pure thread pool implementation, but with async batch run, we also get proper timeout handling.

All Promptflow Contribution checklist:

• The pull request does not introduce [breaking changes].
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.
• Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

promptflow SDK CLI Azure E2E Test Result users/ninhu/async_evaluators

  4 files    4 suites   4m 17s ⏱️
244 tests 204 ✅  40 💤 0 ❌
976 runs  816 ✅ 160 💤 0 ❌

Results for commit 38b77b3.

♻️ This comment has been updated with latest results.

[github-actions]

SDK CLI Global Config Test Result users/ninhu/async_evaluators

6 tests   6 ✅  1m 24s ⏱️
1 suites  0 💤
1 files    0 ❌

Results for commit 38b77b3.

♻️ This comment has been updated with latest results.

[github-actions]

Executor Unit Test Result users/ninhu/async_evaluators

798 tests   798 ✅  3m 46s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 38b77b3.

♻️ This comment has been updated with latest results.

[github-actions]

Executor E2E Test Result users/ninhu/async_evaluators

246 tests   240 ✅  5m 5s ⏱️
  1 suites    6 💤
  1 files      0 ❌

Results for commit 38b77b3.

♻️ This comment has been updated with latest results.

[github-actions]

SDK CLI Test Result users/ninhu/async_evaluators

    4 files      4 suites   1h 6m 43s ⏱️
  786 tests   763 ✅ 23 💤 0 ❌
3 144 runs  3 052 ✅ 92 💤 0 ❌

Results for commit 38b77b3.

♻️ This comment has been updated with latest results.

[weedqian]

Sounds good to me.