Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Changes at 4/16/2022 10:12:41 PM #4874

Closed
performanceautofiler bot opened this issue Apr 26, 2022 · 4 comments
Closed

[Perf] Changes at 4/16/2022 10:12:41 PM #4874

performanceautofiler bot opened this issue Apr 26, 2022 · 4 comments
Labels
arch-x64 branch-refs/heads/main kind-micro os-alpine perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 26, 2022 •
edited
Loading

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ForeachOverList100Elements - Duration of single invocation 6.15 ms 10.51 ms 1.71 0.13 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.ForeachOverList100Elements

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.508131235897439 > 6.460709094218751.
IsChangePoint: Marked as a change because one of 3/15/2022 11:34:41 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -14.468396769996337 (T) = (0 -10275307.406177158) / Math.Sqrt((334158393904.3621 / (46)) + (619183493689.1334 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.55032418984029 = (6627844.339599507 - 10275307.406177158) / 6627844.339599507 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TrimStart - Duration of single invocation 10.28 ns 11.95 ns 1.16 0.18 False
Trim - Duration of single invocation 13.27 ns 14.80 ns 1.11 0.06 False
Trim - Duration of single invocation 10.76 ns 12.05 ns 1.12 0.10 False
Trim - Duration of single invocation 10.55 ns 11.94 ns 1.13 0.14 False

graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.TrimStart(s: " Test")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.950812547755511 > 10.79309906278997.
IsChangePoint: Marked as a change because one of 2/26/2022 3:32:05 AM, 4/2/2022 3:40:16 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -14.889943993101932 (T) = (0 -11.59547902114433) / Math.Sqrt((0.05392207918309299 / (46)) + (0.1117709377822395 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.15835111471837598 = (10.010331818917859 - 11.59547902114433) / 10.010331818917859 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Trim(s: " Te st  ")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.797898187448453 > 13.984379895623395.
IsChangePoint: Marked as a change because one of 2/25/2022 7:19:51 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -35.03087003076377 (T) = (0 -14.786953896362554) / Math.Sqrt((0.08810576309222704 / (46)) + (0.0034671695045072346 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.12598204985964162 = (13.132495227793205 - 14.786953896362554) / 13.132495227793205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Trim(s: "Test ")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.053331851479438 > 11.433688825056453.
IsChangePoint: Marked as a change because one of 2/24/2022 9:06:13 PM, 3/30/2022 6:37:35 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -34.05403580405217 (T) = (0 -12.139535921343105) / Math.Sqrt((0.030276203231365307 / (46)) + (0.012064260147921113 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.13316478079228689 = (10.712948484734389 - 12.139535921343105) / 10.712948484734389 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.Trim(s: " Test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.938674490618524 > 11.200035366850665.
IsChangePoint: Marked as a change because one of 2/25/2022 5:57:13 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -10.472758061236597 (T) = (0 -11.91862347115635) / Math.Sqrt((0.04283604934959887 / (46)) + (0.23237029675301318 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.15008060358461325 = (10.363294045659016 - 11.91862347115635) / 10.363294045659016 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.IndexerSetReverse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IList - Duration of single invocation 1.45 μs 1.79 μs 1.23 0.16 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IndexerSetReverse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSetReverse<String>.IList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7894518972830449 > 1.5267697155497022.
IsChangePoint: Marked as a change because one of 2/17/2022 3:48:53 PM, 2/22/2022 5:41:06 PM, 2/23/2022 9:24:01 AM, 3/17/2022 6:54:53 PM, 3/21/2022 1:28:14 PM, 3/29/2022 3:04:55 PM, 4/7/2022 3:14:09 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -195.83408055766438 (T) = (0 -1788.8803178898627) / Math.Sqrt((126.66793590438694 / (46)) + (1.116326386094395 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.22696089325172697 = (1457.9766378282202 - 1788.8803178898627) / 1457.9766378282202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.TryGetValueTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 9.78 μs 12.01 μs 1.23 0.01 False
IDictionary - Duration of single invocation 10.38 μs 12.24 μs 1.18 0.01 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<String, String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.012891595842882 > 10.333787910923453.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -42.75350088159367 (T) = (0 -11905.650683167747) / Math.Sqrt((29455.27181408835 / (46)) + (15361.080076496253 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.19341150351488232 = (9976.148753470834 - 11905.650683167747) / 9976.148753470834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueTrue&lt;String, String&gt;.IDictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.239603924662124 > 11.164492989068425.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -35.5965631259579 (T) = (0 -12387.271198156424) / Math.Sqrt((15828.184885936735 / (46)) + (25104.080690521205 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.17269885012258407 = (10563.045403226552 - 12387.271198156424) / 10563.045403226552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 8.11 μs 8.88 μs 1.09 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.876139627833837 > 8.577192635751029.
IsChangePoint: Marked as a change because one of 2/3/2022 4:53:52 PM, 2/24/2022 10:17:16 AM, 3/30/2022 6:37:35 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -23.196036177514152 (T) = (0 -8800.573426243192) / Math.Sqrt((4958.130518759598 / (46)) + (6837.403909346844 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.07663830165230878 = (8174.122556049713 - 8800.573426243192) / 8174.122556049713 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 15.60 μs 18.12 μs 1.16 0.20 False
ImmutableArray - Duration of single invocation 16.24 μs 18.22 μs 1.12 0.40 False
Span - Duration of single invocation 14.17 μs 17.16 μs 1.21 0.18 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.Array(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.120819354204603 > 16.47811771003948.
IsChangePoint: Marked as a change because one of 2/12/2022 1:03:18 PM, 2/19/2022 11:51:48 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -3.754539523158522 (T) = (0 -19692.48542669895) / Math.Sqrt((67524.00555975518 / (46)) + (12301418.98113097 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.25274672042341356 = (15719.446800900922 - 19692.48542669895) / 15719.446800900922 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.ImmutableArray(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.216926046511627 > 17.045126678226854.
IsChangePoint: Marked as a change because one of 4/1/2022 5:08:23 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -16.277176572696742 (T) = (0 -18933.13938566031) / Math.Sqrt((387230.7914233105 / (46)) + (323884.6124397518 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.2008921081342669 = (15765.895418427941 - 18933.13938566031) / 15765.895418427941 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsTrue&lt;Int32&gt;.Span(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.162787104166668 > 14.860787836402583.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -56.37745256859708 (T) = (0 -17309.477641302343) / Math.Sqrt((137420.62176312337 / (45)) + (3240.0922790222303 / (11))) is less than -2.0048792881871513 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (45) + (11) - 2, .025) and -0.23223663543621278 = (14047.202577429272 - 17309.477641302343) / 14047.202577429272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.ContainsFalse<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Array - Duration of single invocation 485.89 μs 535.57 μs 1.10 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<String>.Array(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 535.5687054166667 > 510.8680233637456.
IsChangePoint: Marked as a change because one of 2/3/2022 4:53:52 PM, 2/19/2022 11:51:48 PM, 2/23/2022 6:59:06 PM, 2/26/2022 10:27:55 AM, 3/7/2022 10:45:01 PM, 3/14/2022 4:00:26 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -36.68432197211849 (T) = (0 -538645.4987413066) / Math.Sqrt((6740271.210843837 / (46)) + (19317703.938823342 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.10368232184669923 = (488043.9670720069 - 538645.4987413066) / 488043.9670720069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 19.85 μs 21.74 μs 1.09 0.32 False
ICollection - Duration of single invocation 18.27 μs 20.37 μs 1.11 0.34 False
List - Duration of single invocation 17.85 μs 19.15 μs 1.07 0.37 False

graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsFalse<Int32>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.735764957636952 > 20.94848438994739.
IsChangePoint: Marked as a change because one of 2/11/2022 3:03:06 PM, 3/14/2022 9:50:33 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -35.31725186652747 (T) = (0 -21760.64298565448) / Math.Sqrt((125990.85874708007 / (46)) + (694.0362929522713 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.09398596463551001 = (19891.153715947905 - 21760.64298565448) / 19891.153715947905 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.ICollection(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.370499289905013 > 19.210780087584983.
IsChangePoint: Marked as a change because one of 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -30.997225024200286 (T) = (0 -20319.064489712433) / Math.Sqrt((199191.7420539108 / (46)) + (3559.246483247914 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.11615817750249463 = (18204.46680342224 - 20319.064489712433) / 18204.46680342224 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.ContainsFalse&lt;Int32&gt;.List(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.152400795474442 > 18.782756266442927.
IsChangePoint: Marked as a change because one of 2/2/2022 4:36:37 PM, 2/19/2022 11:51:48 PM, 2/22/2022 2:43:53 PM, 4/14/2022 8:50:18 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -9.945159245880799 (T) = (0 -19819.327255541608) / Math.Sqrt((89074.68256826104 / (37)) + (735952.3496783499 / (20))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (20) - 2, .025) and -0.11031653104581432 = (17850.159572850505 - 19819.327255541608) / 17850.159572850505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Tests.Perf_Boolean

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 2.95 ns 4.15 ns 1.41 0.01 False
TryParse - Duration of single invocation 2.95 ns 4.16 ns 1.41 0.04 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_Boolean*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Boolean.TryParse(value: "false")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.148303603474224 > 3.098015135407483.
IsChangePoint: Marked as a change because one of 2/8/2022 11:58:21 PM, 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -1136.0020196335684 (T) = (0 -4.153779206131033) / Math.Sqrt((5.770776123963784E-06 / (46)) + (1.0966556572170448E-05 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.40793943806305827 = (2.950254175595439 - 4.153779206131033) / 2.950254175595439 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Boolean.TryParse(value: "False")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.155272208792124 > 3.099267660750911.
IsChangePoint: Marked as a change because one of 2/3/2022 4:53:52 PM, 2/8/2022 11:58:21 PM, 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -620.2968343528779 (T) = (0 -4.155860483121808) / Math.Sqrt((0.00014166940770667279 / (46)) + (7.555505035106985E-06 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.4078119582643355 = (2.951999703316549 - 4.155860483121808) / 2.951999703316549 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.TryGetValueFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 8.33 μs 10.15 μs 1.22 0.01 False
Dictionary - Duration of single invocation 7.61 μs 9.04 μs 1.19 0.00 True

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.TryGetValueFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueFalse<String, String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.152635144106219 > 8.782365383657185.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -57.398790659994624 (T) = (0 -10085.722866941116) / Math.Sqrt((3745.060874051329 / (46)) + (9210.274335290123 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.2084570769211336 = (8345.950435109522 - 10085.722866941116) / 8345.950435109522 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.TryGetValueFalse&lt;String, String&gt;.Dictionary(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.043817840713052 > 8.010285376607378.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -40.4058070579155 (T) = (0 -9008.503563649987) / Math.Sqrt((5794.528993533562 / (46)) + (10930.488841572553 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.1765857382257541 = (7656.478632177254 - 9008.503563649987) / 7656.478632177254 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 26, 2022 •
edited
Loading

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in Span.Sorting

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BubbleSortArray - Duration of single invocation 144.19 μs 151.71 μs 1.05 0.07 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'Span.Sorting*'

Payloads

Baseline
Compare

Histogram

Span.Sorting.BubbleSortArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 151.70555542857144 > 151.386540725.
IsChangePoint: Marked as a change because one of 3/6/2022 1:47:20 PM, 3/9/2022 2:35:55 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -32.754521014946164 (T) = (0 -151751.25782001333) / Math.Sqrt((3227947.4514640383 / (46)) + (4282.860215508543 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.060822978992230395 = (143050.5002485667 - 151751.25782001333) / 143050.5002485667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.IterateFor<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ImmutableArray - Duration of single invocation 116.78 ns 226.90 ns 1.94 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateFor&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateFor<String>.ImmutableArray(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 226.9007376447516 > 122.6767321460387.
IsChangePoint: Marked as a change because one of 2/23/2022 6:59:06 PM, 2/26/2022 10:27:55 AM, 3/3/2022 10:59:13 PM, 3/11/2022 8:42:43 PM, 3/21/2022 1:28:14 PM, 3/29/2022 3:04:55 PM, 4/7/2022 3:14:09 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -2760.9086677895084 (T) = (0 -226.92607970236824) / Math.Sqrt((0.06603185550383836 / (46)) + (0.0017012101934099125 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.9423469705041189 = (116.83086654876682 - 226.92607970236824) / 116.83086654876682 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Numerics.Tests.Perf_BitOperations

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LeadingZeroCount_uint - Duration of single invocation 371.87 ns 439.94 ns 1.18 0.02 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_BitOperations*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_BitOperations.LeadingZeroCount_uint

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 439.9361941986373 > 391.12170897867327.
IsChangePoint: Marked as a change because one of 2/3/2022 4:53:52 PM, 3/15/2022 11:34:41 AM, 3/16/2022 12:48:00 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -333.2420629413616 (T) = (0 -439.8230492527828) / Math.Sqrt((0.1248576981618643 / (46)) + (0.42290696743085754 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.18163764656359158 = (372.2148245122902 - 439.8230492527828) / 372.2148245122902 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in GuardedDevirtualization.TwoClassVirtual

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Call - Duration of single invocation 3.75 ns 4.77 ns 1.27 0.05 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'GuardedDevirtualization.TwoClassVirtual*'

Payloads

Baseline
Compare

Histogram

GuardedDevirtualization.TwoClassVirtual.Call(testInput: pB = 0.50)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.76838906128668 > 3.9371540223881687.
IsChangePoint: Marked as a change because one of 2/19/2022 11:51:48 PM, 2/26/2022 10:27:55 AM, 3/7/2022 10:45:01 PM, 3/17/2022 6:54:53 PM, 3/31/2022 1:45:25 PM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -151.29650514674873 (T) = (0 -4.765355218038869) / Math.Sqrt((0.001989189951303827 / (46)) + (2.0627019955910152E-05 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.2710689558754346 = (3.7490926011616605 - 4.765355218038869) / 3.7490926011616605 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
List - Duration of single invocation 284.33 ns 513.05 ns 1.80 0.01 False
ConcurrentQueue - Duration of single invocation 2.94 μs 3.28 μs 1.11 0.09 False

graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IterateForEach<Int32>.List(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 513.0496577748503 > 298.78029402071047.
IsChangePoint: Marked as a change because one of 3/15/2022 12:53:39 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -2463.031515644672 (T) = (0 -512.8628721075002) / Math.Sqrt((0.05548813997634782 / (46)) + (0.08103532892285802 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.8007315182353199 = (284.8080721162116 - 512.8628721075002) / 284.8080721162116 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.IterateForEach&lt;Int32&gt;.ConcurrentQueue(Size: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2787198077499626 > 3.087562155516451.
IsChangePoint: Marked as a change because one of 3/4/2022 3:52:45 PM, 3/21/2022 1:28:14 PM, 3/29/2022 3:04:55 PM, 3/31/2022 1:45:25 PM, 4/14/2022 8:50:18 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -332.3100898273043 (T) = (0 -3283.8440983883074) / Math.Sqrt((5.550079904278217 / (37)) + (18.319366785931713 / (20))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (37) + (20) - 2, .025) and -0.11666977225603364 = (2940.7477304179924 - 3283.8440983883074) / 2940.7477304179924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS alpine 3.12
Baseline 2c3e1c7fee778b1e58c0678eb9bac273fd700216
Compare a39ddca87f9d8e99d74ef44f280761bde578aca2
Diff Diff

Regressions in System.Collections.Concurrent.AddRemoveFromDifferentThreads<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentQueue - Duration of single invocation 24.06 ms 32.05 ms 1.33 0.52 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.Concurrent.AddRemoveFromDifferentThreads&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Concurrent.AddRemoveFromDifferentThreads<String>.ConcurrentQueue(Size: 2000000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.0515883 > 24.280855423125.
IsChangePoint: Marked as a change because one of 2/24/2022 12:54:34 PM, 3/14/2022 9:01:19 AM, 4/16/2022 5:36:25 PM, 4/19/2022 4:39:14 PM falls between 4/10/2022 11:44:05 PM and 4/19/2022 4:39:14 PM.
IsRegressionStdDev: Marked as regression because -6.6120056213276825 (T) = (0 -31806596.828468908) / Math.Sqrt((18943918669435.004 / (46)) + (9871111561570.65 / (11))) is less than -2.0040447832881556 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (46) + (11) - 2, .025) and -0.31209232448600166 = (24241127.118039355 - 31806596.828468908) / 24241127.118039355 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added alpine 3.12 PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Apr 26, 2022
@AndyAyersMS
Copy link
Member

PGO

@dakersnar
Copy link

More regressions detected on System.Collections.IterateForEach.List(Size: 512)

@AndyAyersMS I'm assuming these regressions are "expected" and no followup is needed?

System.Collections.IterateForEach.List(Size: 512)

Result Ratio Operating System Bit Processor Name
Slower 0.80 ubuntu 20.04 Arm64 Unknown processor
Same 1.00 Windows 10 Arm64 Microsoft SQ1 3.0 GHz
Same 1.00 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 1.00 Windows 11 Arm64 Unknown processor
Slower 0.68 macOS Monterey 12.3 Arm64 Apple M1 Max
Same 1.00 Windows 10 X64 Intel Xeon Platinum 8272CL CPU 2.60GHz
Same 0.99 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 1.01 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 1.00 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 1.01 Windows 11 X64 AMD Ryzen 9 5950X
Same 1.00 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.99 Windows 11 X64 Intel Core i9-9900T CPU 2.10GHz
Slower 0.56 alpine 3.13 X64 Intel Core i7-7700 CPU 3.60GHz (Kaby Lake)
Slower 0.64 pop 22.04 X64 Intel Core i7-6600U CPU 2.60GHz (Skylake)
Slower 0.89 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Slower 0.80 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.52 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.63 ubuntu 20.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 1.00 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 1.01 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 0.99 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.89 macOS Monterey 12.2.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Slower 0.88 macOS Monterey 12.3.1 X64 Intel Core i7-4870HQ CPU 2.50GHz (Haswell)

@AndyAyersMS
Copy link
Member

I should probably investigate this one, it seems persistent.

@dakersnar dakersnar mentioned this issue May 16, 2022
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro os-alpine perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-coreclr
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AndyAyersMS @dakersnar