Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Regression on 8/1/2022 6:03:22 PM #73630

Closed
performanceautofiler bot opened this issue Aug 9, 2022 · 6 comments
Closed

[Perf] Regression on 8/1/2022 6:03:22 PM #73630

performanceautofiler bot opened this issue Aug 9, 2022 · 6 comments
Assignees
Labels
Milestone

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline f1de614c74589fe8f682dda39889d189f2522504
Compare 68bd58e0e68b22a3cbcc67a3f1218a6c8f623bcf
Diff Diff

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 142.23 ns 162.00 ns 1.14 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: False)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 162.0041171183368 > 140.82233369196862.
IsChangePoint: Marked as a change because one of 6/8/2022 9:45:10 AM, 6/23/2022 2:33:56 PM, 8/1/2022 4:16:36 PM, 8/5/2022 10:47:59 PM falls between 7/28/2022 9:13:34 AM and 8/5/2022 10:47:59 PM.
IsRegressionStdDev: Marked as regression because -16.15588523243305 (T) = (0 -155.80685151415716) / Math.Sqrt((18.262167754812662 / (46)) + (12.77726289932357 / (28))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (28) - 2, .025) and -0.10593337674114203 = (140.88267412027457 - 155.80685151415716) / 140.88267412027457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Aug 9, 2022
@DrewScoggins DrewScoggins removed untriaged New issue has not been triaged by the area owner refs/heads/main labels Aug 9, 2022
@DrewScoggins DrewScoggins transferred this issue from dotnet/perf-autofiling-issues Aug 9, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Aug 9, 2022
@DrewScoggins DrewScoggins added x64 os-windows tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark arch-x64 and removed untriaged New issue has not been triaged by the area owner x64 labels Aug 9, 2022
@DrewScoggins
Copy link
Member

Seems related to #72923 @mrsharm

@ghost
Copy link

ghost commented Aug 10, 2022

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.19042
Baseline f1de614c74589fe8f682dda39889d189f2522504
Compare 68bd58e0e68b22a3cbcc67a3f1218a6c8f623bcf
Diff Diff

Regressions in System.Tests.Perf_GC<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
AllocateUninitializedArray - Duration of single invocation 142.23 ns 162.00 ns 1.14 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Char&gt;*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_GC<Char>.AllocateUninitializedArray(length: 10000, pinned: False)


Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 162.0041171183368 > 140.82233369196862.
IsChangePoint: Marked as a change because one of 6/8/2022 9:45:10 AM, 6/23/2022 2:33:56 PM, 8/1/2022 4:16:36 PM, 8/5/2022 10:47:59 PM falls between 7/28/2022 9:13:34 AM and 8/5/2022 10:47:59 PM.
IsRegressionStdDev: Marked as regression because -16.15588523243305 (T) = (0 -155.80685151415716) / Math.Sqrt((18.262167754812662 / (46)) + (12.77726289932357 / (28))) is less than -1.9934635666612892 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (28) - 2, .025) and -0.10593337674114203 = (140.88267412027457 - 155.80685151415716) / 140.88267412027457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: dakersnar
Labels:

os-windows, tenet-performance, tenet-performance-benchmarks, arch-x64, area-GC-coreclr

Milestone: -

@mangod9
Copy link
Member

mangod9 commented Aug 10, 2022

The change seems to be within the typical range for this particular test?

@mangod9 mangod9 added this to the 8.0.0 milestone Aug 10, 2022
@mrsharm
Copy link
Member

mrsharm commented Aug 10, 2022

I agree with @mangod9, I think this issue might be noisy with respect to #72923.

I merged the PR ~9:30 AM PST for which the next point in the trend seemed to be at a similar previous level:

image

Consequent regressions took place after the commit with the inflection point being on 8/3:

image

It seems like the moving average also indicates similar behavior:

image

Could this be related to some other commit?

@mangod9
Copy link
Member

mangod9 commented Aug 2, 2023

Closing this older Regions related issues, since they have been investigated.

@mangod9 mangod9 closed this as completed Aug 2, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

5 participants