[PromptFlow] Optimize memory computation

[Hhhilulu]

Description

Optimize memory computation

1. Optimize the calculation logic of the maximum number of processes
2. Prioritize using the environment variable "PF_WORKER_COUNT" as the number of processes if users set.

The following table summarizes the logic for determining the number of processes and corresponding log messages under different conditions (based on whether fork is used and the setting of the environment variable PF_WORKER_COUNT):

Scenario	Fork_Used	PF_WORKER_COUNT Set	Worker Count	NLines	Estimated Available Worker Count	Calculation of Processes
1	No(spawn)	No	16	4	2	Min(16, 4, 2) = 2
Log Message: Set process count to 2 by taking the minimum value among the factors of {'default_worker_count': 16, 'row_count': 4, 'estimated_worker_count_based_on_memory_usage': 2}.
2	No(spawn)	Yes (2)	2	4	2	PF_WORKER_COUNT = 2
Log Message: Set process count to 2 with the environment variable 'PF_WORKER_COUNT'.
3	No(spawn)	Yes (6)	6	4	2	PF_WORKER_COUNT = 6
Log Message: Set process count to 6 with the environment variable 'PF_WORKER_COUNT'. Warning Message: The current process count (6) is larger than recommended process count (2) that estimated by system available memory. This may cause memory exhaustion.
4	Yes(fork)	No	16	4	N/A	Min(16, 4) = 4
Log Message: Set process count to 4 by taking the minimum value among the factors of {'default_worker_count': 16, 'row_count': 4}.
5	Yes(fork)	Yes (3)	3	4	N/A	PF_WORKER_COUNT = 3
Log Message: Set process count to 3 with the environment variable 'PF_WORKER_COUNT'.

All Promptflow Contribution checklist:

• The pull request does not introduce [breaking changes].
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.
• Create an issue and link to the pull request to get dedicated review from promptflow team. Learn more: suggested workflow.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

SDK CLI Global Config Test Result luluzuo/optimize_memory_computation

2 tests   2 ✔️  41s ⏱️
1 suites  0 💤
1 files    0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.

[github-actions]

Executor E2E Test Result luluzuo/optimize_memory_computation

146 tests   144 ✔️  1m 43s ⏱️
    1 suites      2 💤
    1 files        0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.

[github-actions]

promptflow SDK CLI Azure E2E Test Result luluzuo/optimize_memory_computation

    2 files      2 suites   1m 24s ⏱️
114 tests   95 ✔️ 19 💤 0 ❌
228 runs  190 ✔️ 38 💤 0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.

[github-actions]

Executor Unit Test Result luluzuo/optimize_memory_computation

595 tests   590 ✔️  53s ⏱️
    1 suites      5 💤
    1 files        0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.

[github-actions]

SDK PFS E2E Test Result luluzuo/optimize_memory_computation

  2 files    2 suites   1m 18s ⏱️
15 tests 15 ✔️ 0 💤 0 ❌
30 runs  30 ✔️ 0 💤 0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.

[github-actions]

SDK CLI Test Result luluzuo/optimize_memory_computation

0 tests   0 ✔️  0s ⏱️
0 suites  0 💤
0 files    0 ❌

Results for commit c1e228a.

♻️ This comment has been updated with latest results.