Enable executor recording mode

[chw-microsoft]

Description

This PR is focused on enabling the recording injection mode for CI/tests in the execution environment. The recording mode provides a mechanism to record and replay end-to-end tests, enhancing the reliability and efficiency of our testing process.

For detailed information on the recording mode, please refer to our documentation.

Key Features of This PR:

Generation of Test Records

The generation of records for execution tests when they are missing in the shelve database, to make it work under recording mode.

Multi-Process Compatibility

Resolves issues related to the recording injection mode in a multi-process environment, particularly with processes spawned/forkserver.
What is the issue under multi/new Process(spawn/forkserver)?
Spawn/forkserver mode will not replicate the resources/setups from main Process, including the recording setup. This would make the recording not working anymore.

How you resolved this issue?
There are multiple ways to pass the recording setup in new Process, like environment variable, serializable object as argument etc. But these might incur interface change or too complex to squeeze into simple state object.
We choose to re-import the state into new Process.

1. Create new target method for Process: this new target is used to re-import the state needed

Example: For new Process target method _process_wrapper, for define another wrapper method outside and inject the recording state

2. Define a customized Process class with above new targets

Enable this new target method whenever new Process spawned/forkservered

3. Override context.Process or multiprocess.Process class

        multiprocessing.get_context("spawn").Process = MockSpawnProcess
   

or

        multiprocessing.Process = MockSpawnProcess

We have implemented above logic in codes and integrated as part of recording injection fixture for testing.

So all the tests under executor/sdk would intercept the third-party call like this as default in CI?

Yes, all the CI is enable with "PROMPT_FLOW_TEST_MODE=replay". All the openai/aoai related calls would be mocked to do recording result retrieval rather than real api request.

Sometimes we might have necessity to customize the openai/AOAI call in tests. How shall we do instead of using the default recording mock?

Yes,

1. Create your own target with customized mocking about openai/aoai call .
2. Override the default recording target via context manager "override_process_target"

Sample Tests: test_executor_openai_telemetry

In which scope the recording mode is enabled?
The recording mode is enabled per requiriment. If the test are involved with 3rd party service with connection, like openai/aoai etc, it is required to enable recording mode. If not, your PR will fail at the CI tests since the connections info are not configured.
To enable recording mode, just make sure the fixture "recording_injection" is required in your test class level or method level.

Why not make recording mode a session-level fixture or required fixture for all tests?

1. recording setup is complicated. it might introduce expected behavior if abuse
2. Some tests like test_executor_openai_telemetry might server some special test purpose. If enable recording, the customized mocking might be not easy to be configured.

Note:
Above logic resolved this issue for one layer of new Processing. If you have nested new Processing action , you need to repeat above logic at each layer.

Todos and Ongoing Work[updated]:

[Done]Metrics Mocks

Currently, this openai_injector is skipped as it does not involve actual openai/aoai requests. This omission leads to potential inaccuracies in metrics calculation, especially token usage, in recording mode. Future work will focus on integrating openai_injector.py into the recording mode.
--- this fixed already by [fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)

[Todo]Consolidation of Configuration

Efforts are underway to consolidate the configuration settings for recording injection in both the execution environment and the SDK, aiming for a more unified and streamlined setup.

[Done]Record error info

Record not only regular tool execution result, but also error response if exception ocurr
----- have fixed this in this PR, sample tests: test_executor_node_overrides

[Done]Test langchain

Support langchain test with agent

---- fixed by [fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)

All Promptflow Contribution checklist:

• The pull request does not introduce [breaking changes].
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.
• Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

Executor Unit Test Result

705 tests   705 ✅  50s ⏱️
  1 suites    0 💤
  1 files      0 ❌

Results for commit 30b4dd2.

[github-actions]

Hi, thank you for your interest in helping to improve the prompt flow experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment.