[fundamental] Recording support metrics #1762
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SDK CLI Global Config Test Result yigao/recorder_refactor2 tests 2 ✅ 44s ⏱️ Results for commit c4045e3. ♻️ This comment has been updated with latest results. |
Executor Unit Test Result yigao/recorder_refactor700 tests 700 ✅ 43s ⏱️ Results for commit c4045e3. ♻️ This comment has been updated with latest results. |
promptflow SDK CLI Azure E2E Test Result yigao/recorder_refactor 4 files 4 suites 3m 21s ⏱️ Results for commit c4045e3. ♻️ This comment has been updated with latest results. |
Executor E2E Test Result yigao/recorder_refactor190 tests 188 ✅ 3m 58s ⏱️ Results for commit c4045e3. ♻️ This comment has been updated with latest results. |
SDK CLI Test Result yigao/recorder_refactor 12 files 12 suites 38m 24s ⏱️ Results for commit c4045e3. ♻️ This comment has been updated with latest results. |
crazygao
force-pushed
the
yigao/recorder_refactor
branch
from
61b3022 to
31b4c34
Compare
github-actions
bot
added
the
executor
zhengfeiwang
previously approved these changes
Jan 30, 2024
wangchao1230
previously approved these changes
Jan 30, 2024
crazygao
dismissed stale reviews from wangchao1230 and zhengfeiwang
via
c4045e3
chw-microsoft
approved these changes
Jan 30, 2024
zhengfeiwang
approved these changes
Jan 30, 2024
7 tasks
7 tasks
chw-microsoft
added a commit
that referenced
this pull request
Mar 8, 2024
# Description This PR is focused on enabling the recording injection mode for CI/tests in the execution environment. The recording mode provides a mechanism to record and replay end-to-end tests, enhancing the reliability and efficiency of our testing process. For detailed information on the recording mode, please refer to our [documentation](https://github.com/microsoft/promptflow/blob/main/docs/dev/replay-e2e-test.md). In this PR, we still keep the connection configuration in CI to make sure backward compatibilities. Meanwhile, enable the recording injection fixture to switch test into recording mode. ## **Key Features of This PR:** ### **Generation of Test Records** The generation of records for execution tests when they are missing in the shelve database, to make it work under recording mode. ### **Multi-Process Compatibility** Resolves issues related to the recording injection mode in a multi-process environment, particularly with processes spawned/forkserver. **_What is the issue under multi/new Process(spawn/forkserver)?_** Spawn/forkserver mode will not replicate the resources/setups from main Process, including the recording setup. This would make the recording not working anymore.  **_How you resolved this issue?_** There are multiple ways to pass the recording setup in new Process, like environment variable, serializable object as argument etc. But these might incur interface change or too complex to squeeze into simple state object. We choose to re-import the state into new Process. 1) Create new target method for Process: this new target is used to re-import the state needed Example: For new Process target method _process_wrapper, for define another wrapper method outside and inject the recording state  2) Define a customized Process class with above new targets Enable this new target method whenever new Process spawned/forkservered  3) Override context.Process or multiprocess.Process class multiprocessing.get_context("spawn").Process = MockSpawnProcess or multiprocessing.Process = MockSpawnProcess We have implemented above logic in codes and integrated as part of recording injection fixture for testing. **_So all the tests under executor/sdk would intercept the third-party call like this as default in CI?_** Yes, all the CI is enable with "PROMPT_FLOW_TEST_MODE=replay". All the openai/aoai related calls would be mocked to do recording result retrieval rather than real api request. **_Sometimes we might have necessity to customize the openai/AOAI call in tests. How shall we do instead of using the default recording mock?_** Yes, 1) Create your own target with customized mocking about openai/aoai call . 2) Override the default recording target via context manager "override_process_target" Sample Tests: test_executor_openai_telemetry **_In which scope the recording mode is enabled?_** The recording mode is enabled per requiriment. If the test are involved with 3rd party service with connection, like openai/aoai etc, it is required to enable recording mode. If not, your PR will fail at the CI tests since the connections info are not configured. To enable recording mode, just make sure the fixture "recording_injection" is required in your test class level or method level. _**Why not make recording mode a session-level fixture or required fixture for all tests?**_ 1) recording setup is complicated. it might introduce expected behavior if abuse 2) Some tests like **test_executor_openai_telemetry** might server some special test purpose. If enable recording, the customized mocking might be not easy to be configured. **_Note:_** _Above logic resolved this issue for one layer of new Processing. If you have nested new Processing action , you need to repeat above logic at each layer._ ## **Todos and Ongoing Work[updated]:** ### **[Done]Metrics Mocks** Currently, this openai_injector is skipped as it does not involve actual openai/aoai requests. This omission leads to potential inaccuracies in metrics calculation, especially token usage, in recording mode. Future work will focus on integrating openai_injector.py into the recording mode. --- this fixed already by [[fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)](#1762) ### **[Todo]Consolidation of Configuration** Efforts are underway to consolidate the configuration settings for recording injection in both the execution environment and the SDK, aiming for a more unified and streamlined setup. ### **[Done]Record error info** Record not only regular tool execution result, but also error response if exception ocurr ----- have fixed this in this PR, sample tests: test_executor_node_overrides ### **[Done]Test langchain** Support langchain test with agent ---- fixed by [[fundamental] Recording support metrics by crazygao · Pull Request #1762 · microsoft/promptflow (github.com)](#1762) ### **[Todo] Create pipeline CI without connection to monitor** We need to CI without connection to identify tests which needs the record setup. ### **[Todo] Make migration plan to make the non-connection CI as main CI** Need to make plan to safely switch to new record mode. Make sure there are fallback logic if record has flow design which might hinder the new feature delivery. ### **[Todo] Enable more apis with recording mode** like assistant apis & tests # All Promptflow Contribution checklist: - [x] **The pull request does not introduce [breaking changes].** - [ ] **CHANGELOG is updated for new features, bug fixes or other significant changes.** - [x] **I have read the [contribution guidelines](../CONTRIBUTING.md).** - [x] **Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: [suggested workflow](../CONTRIBUTING.md#suggested-workflow).** ## General Guidelines and Best Practices - [x] Title of the pull request is clear and informative. - [ ] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, [see this page](https://github.com/Azure/azure-powershell/blob/master/documentation/development-docs/cleaning-up-commits.md). ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. --------- Co-authored-by: Philip Gao <[email protected]> Co-authored-by: chjinche <[email protected]> Co-authored-by: Cheng Liu <[email protected]> Co-authored-by: Han Wang <[email protected]> Co-authored-by: chenslucky <[email protected]> Co-authored-by: cs_lucky <[email protected]> Co-authored-by: Ying Chen <[email protected]> Co-authored-by: Ying Chen <[email protected]> Co-authored-by: chenyang <[email protected]> Co-authored-by: Peiwen Gao <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
How to support metrics:
Old injection inject mock_tool, and all logic below mock_tool is dropped. especially for metric collection
Good point. mock_tool could also record items with no llm calls.
New way:
add a new layer at openai_inject.
In this way, the logic below mock_tool is not affected. For openai calls, it is easily collected.
Bad point, it doesn't know other info besides openai call.
So the new recording is a combination of these two.
Inject openai calls for llm nodes. + Inject tool for python nodes.
Also upload token numbers in live mode. Here is an example pipeline
https://github.com/microsoft/promptflow/actions/runs/7705912004
And we can retrieve the uploaded artifact.
All Promptflow Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines