Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[EP Perf] Add concurrency test #19804

Merged
merged 16 commits into from
Mar 15, 2024
Merged

[EP Perf] Add concurrency test #19804

merged 16 commits into from
Mar 15, 2024

Conversation

yf711
Copy link
Contributor

@yf711 yf711 commented Mar 6, 2024 •
edited
Loading

Description

  • Add concurrency test to EP Perf CI panel (impl. by onnx_test_runner)

image

  • Model: FasterRCNN-10 model within CI image
  • -c param configurable via CI panel when kicking off CI tasks
  • Auto-replicate test input/outputs according to -c param
    • By default, the model test will be executed in 100 iterations (~2min added to T4 CI task load overall)

Motivation and Context

To monitor potential concurrency issues of ORT-TRT

@yf711 yf711 marked this pull request as ready for review March 14, 2024 23:23
@yf711 yf711 requested a review from a team as a code owner March 14, 2024 23:23
@yf711 yf711 requested a review from jywu-msft March 14, 2024 23:24
@yf711 yf711 merged commit 0b2a75b into main Mar 15, 2024
91 of 96 checks passed
@yf711 yf711 deleted the yifanl/ep_perf_test_runner branch March 15, 2024 14:41
yf711 added a commit that referenced this pull request Apr 29, 2024
@yf711
Revert "[EP Perf] Add concurrency test (#19804)"
827d96d 
This reverts commit 0b2a75b.
yf711 added a commit that referenced this pull request Apr 30, 2024
@yf711
Revert "[EP Perf] Add concurrency test (#19804)"
90962b3 
This reverts commit 0b2a75b.
yf711 added a commit that referenced this pull request May 2, 2024
@yf711
Partially revert "[EP Perf] Add concurrency test (#19804)"
602f54a 
This partially reverts commit 0b2a75b.
@yf711 yf711 mentioned this pull request May 15, 2024
Merged
yf711 added a commit that referenced this pull request May 16, 2024
@yf711
[EP Perf] Fix on EP Perf (#20683)
47a178b 
### Description
<!-- Describe your changes. -->
* Partially revert [previous
change](#19804), and
   * Redo concurrency_test_result parser outside of post.py
* Add support of syncing memtest result to db


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
To fix the error when CI is running on two model groups.
- When running on two model groups, the [previous
change](#19804) wrongly
navigates two levels up in the directory after running one model group,
while one level is needed. After that, the script can't find another
model group.
- Running on one model group can't repro the issue
yf711 added a commit that referenced this pull request Jun 21, 2024
@yf711
[EP Perf] Fix on EP Perf (#20683)
89d796a 
### Description
<!-- Describe your changes. -->
* Partially revert [previous
change](#19804), and
   * Redo concurrency_test_result parser outside of post.py
* Add support of syncing memtest result to db


### Motivation and Context
<!-- - Why is this change required? What problem does it solve?
- If it fixes an open issue, please link to the issue here. -->
To fix the error when CI is running on two model groups.
- When running on two model groups, the [previous
change](#19804) wrongly
navigates two levels up in the directory after running one model group,
while one level is needed. After that, the script can't find another
model group.
- Running on one model group can't repro the issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jywu-msft jywu-msft jywu-msft approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@yf711 @jywu-msft