Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Tests.Perf_UInt64 #71771

Closed
performanceautofiler bot opened this issue Jun 30, 2022 · 8 comments
Closed

Regressions in System.Tests.Perf_UInt64 #71771

performanceautofiler bot opened this issue Jun 30, 2022 · 8 comments
Assignees
@dakersnar
Labels
arch-arm64 area-System.Runtime runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
7.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 348.38 μs 384.99 μs 1.11 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444.
IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseHex - Duration of single invocation 4.52 ns 5.85 ns 1.30 0.35 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryParseHex(value: "0")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719.
IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsDefinedMethodOverrideMiss - Duration of single invocation 660.32 ns 694.28 ns 1.05 0.18 False
IsDefinedClassMissInherit - Duration of single invocation 1.11 μs 1.21 μs 1.09 0.13 False
IsDefinedMethodOverrideHit - Duration of single invocation 654.04 ns 705.29 ns 1.08 0.15 False
IsDefinedMethodBaseHitInherit - Duration of single invocation 660.85 ns 701.58 ns 1.06 0.16 False
IsDefinedClassHit - Duration of single invocation 606.11 ns 673.82 ns 1.11 0.19 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Attributes.IsDefinedMethodOverrideMiss

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassMissInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Jun 30, 2022
@dakersnar dakersnar transferred this issue from dotnet/perf-autofiling-issues Jul 7, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@dakersnar dakersnar changed the title [Perf] Changes at 6/24/2022 10:21:49 PM Regressions in System.Tests.Perf_UInt64 Jul 7, 2022
@dakersnar
Copy link
Contributor

dakersnar commented Jul 7, 2022 •
edited
Loading

@stephentoub regression in that second benchmark, we think it's caused by #71146. Based on the PR this might be an expected regression.

@dakersnar dakersnar added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Jul 7, 2022
@ghost
Copy link

ghost commented Jul 8, 2022

Tagging subscribers to this area: @dotnet/area-system-globalization
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 348.38 μs 384.99 μs 1.11 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444.
IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseHex - Duration of single invocation 4.52 ns 5.85 ns 1.30 0.35 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryParseHex(value: "0")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719.
IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsDefinedMethodOverrideMiss - Duration of single invocation 660.32 ns 694.28 ns 1.05 0.18 False
IsDefinedClassMissInherit - Duration of single invocation 1.11 μs 1.21 μs 1.09 0.13 False
IsDefinedMethodOverrideHit - Duration of single invocation 654.04 ns 705.29 ns 1.08 0.15 False
IsDefinedMethodBaseHitInherit - Duration of single invocation 660.85 ns 701.58 ns 1.06 0.16 False
IsDefinedClassHit - Duration of single invocation 606.11 ns 673.82 ns 1.11 0.19 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Attributes.IsDefinedMethodOverrideMiss

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassMissInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: dakersnar
Labels:

area-System.Globalization, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.19041, Regression, CoreClr, arm64
Milestone: -

@stephentoub
Copy link
Member

@stephentoub regression in that second benchmark, we think it's caused by #71146. Based on the PR this might be an expected regression.

I'd be surprised if it directly contributed... the method that changed shouldn't be invoked in that benchmark (even though it's in the right area of the code)... it's a non-inlineable method that's only used if there are trailing nulls, which this test doesn't have.

I'll take a look, though.

@ghost
Copy link

ghost commented Jul 8, 2022

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SortedList - Duration of single invocation 348.38 μs 384.99 μs 1.11 0.03 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Collections.ContainsKeyFalse&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<String, String>.SortedList(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 384.9944918699187 > 366.0416569444444.
IsChangePoint: Marked as a change because one of 6/3/2022 8:17:54 AM, 6/6/2022 3:15:19 PM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -18.592276578561822 (T) = (0 -386239.64188622445) / Math.Sqrt((827784.6548322291 / (25)) + (73255526.55193321 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.10803726811218585 = (348580.00989829405 - 386239.64188622445) / 348580.00989829405 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParseHex - Duration of single invocation 4.52 ns 5.85 ns 1.30 0.35 False

graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryParseHex(value: "0")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.851650109031447 > 5.099529018869719.
IsChangePoint: Marked as a change because one of 3/7/2022 12:49:56 AM, 3/7/2022 10:45:01 PM, 3/12/2022 3:27:04 PM, 3/13/2022 11:50:20 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -22.045017915913398 (T) = (0 -5.800006004212765) / Math.Sqrt((0.03388441533165363 / (24)) + (0.012577560483055143 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.2115580874843213 = (4.7872289939113815 - 5.800006004212765) / 4.7872289939113815 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS Windows 10.0.19041
Baseline b92de6bf0351280cd36221f3232b2964a4e61e88
Compare d4a9ade2dfbee1ef532e7793ea9c330c51b5c028
Diff Diff

Regressions in System.Reflection.Attributes

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsDefinedMethodOverrideMiss - Duration of single invocation 660.32 ns 694.28 ns 1.05 0.18 False
IsDefinedClassMissInherit - Duration of single invocation 1.11 μs 1.21 μs 1.09 0.13 False
IsDefinedMethodOverrideHit - Duration of single invocation 654.04 ns 705.29 ns 1.08 0.15 False
IsDefinedMethodBaseHitInherit - Duration of single invocation 660.85 ns 701.58 ns 1.06 0.16 False
IsDefinedClassHit - Duration of single invocation 606.11 ns 673.82 ns 1.11 0.19 False

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Reflection.Attributes*'

Payloads

Baseline
Compare

Histogram

System.Reflection.Attributes.IsDefinedMethodOverrideMiss

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 694.281147282633 > 668.6631138733229.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -8.451246977120563 (T) = (0 -708.0806406154733) / Math.Sqrt((495.03377189893916 / (25)) + (567.5766624145348 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.09351064462495992 = (647.5297191627457 - 708.0806406154733) / 647.5297191627457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassMissInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2112155248820418 > 1.13694964638556.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -7.745527387390561 (T) = (0 -1213.9608926620335) / Math.Sqrt((1954.7879162499878 / (24)) + (1039.2474975435828 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08140105517635117 = (1122.581568467274 - 1213.9608926620335) / 1122.581568467274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodOverrideHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 705.2855175216763 > 693.5954236978833.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.1016749220948405 (T) = (0 -709.1377152585584) / Math.Sqrt((647.7044138314708 / (25)) + (994.2426132288017 / (18))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (18) - 2, .025) and -0.08402020507970037 = (654.1738907960857 - 709.1377152585584) / 654.1738907960857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedMethodBaseHitInherit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 701.575127475433 > 696.2912043948237.
IsChangePoint: Marked as a change because one of 3/17/2022 6:54:53 PM, 5/9/2022 6:32:22 AM, 6/24/2022 5:32:42 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.276517735042958 (T) = (0 -709.118067717889) / Math.Sqrt((788.1459480483514 / (24)) + (809.2232784828515 / (18))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (18) - 2, .025) and -0.08468007639528252 = (653.7578067023238 - 709.118067717889) / 653.7578067023238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Reflection.Attributes.IsDefinedClassHit

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 673.8237605760315 > 640.9646320332743.
IsChangePoint: Marked as a change because one of 5/9/2022 6:32:22 AM, 6/22/2022 11:10:17 PM, 6/30/2022 12:21:07 PM falls between 6/21/2022 9:29:01 PM and 6/30/2022 12:21:07 PM.
IsRegressionStdDev: Marked as regression because -6.410882182847598 (T) = (0 -669.5355188242203) / Math.Sqrt((867.0339451611512 / (17)) + (709.075798026562 / (26))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (26) - 2, .025) and -0.09255424522176141 = (612.8167290113098 - 669.5355188242203) / 612.8167290113098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: dakersnar
Labels:

area-System.Runtime, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.19041, Regression, CoreClr, arm64
Milestone: -

@dakersnar
Copy link
Contributor

Another similar regression dotnet/perf-autofiling-issues#6494

@jeffhandley jeffhandley added this to the 7.0.0 milestone Aug 2, 2022
@ghost ghost removed the untriaged New issue has not been triaged by the area owner label Aug 2, 2022
@dakersnar
Copy link
Contributor

We have no reasonable steps to fix this regression, and it is on the nanosecond scale, so the impact is low. Closing this.

@mrsharm
Copy link
Member

mrsharm commented Aug 12, 2022

A similar regression for showed up for the Windows x64 configuration in our August Performance report and has been characterized as "by design":

image

System.Tests.Perf_UInt64.ParseSpan(value: "18446744073709551615")

Result Ratio Alloc Delta Operating System Bit Processor Name
Same 1.00 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Same 0.98 +0 Windows 11 Arm64 Microsoft SQ1 3.0 GHz
Slower 0.85 +0 macOS Monterey 12.3 Arm64 Apple M1 Max
Slower 0.85 +0 Windows 10 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 0.92 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Faster 1.12 +0 Windows 10 X64 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 1.10 +0 Windows 10 X64 Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R)
Slower 0.85 +0 Windows 10 X64 Intel Core i9-10900K CPU 3.70GHz
Same 1.00 +0 Windows 11 X64 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Same 0.96 +0 Windows 11 X64 AMD Ryzen 9 3950X
Slower 0.86 +0 Windows 11 X64 AMD Ryzen 9 5900X
Same 1.08 +0 Windows 11 X64 AMD Ryzen 9 5950X
Same 1.07 +0 Windows 11 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Same 0.91 +0 Windows 11 X64 Intel Core i9-10900K CPU 3.70GHz
Same 0.91 +0 Windows 11 X64 11th Gen Intel Core i9-11900H 2.50GHz
Slower 0.90 +0 ubuntu 18.04 X64 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 1.01 +0 ubuntu 18.04 X64 Intel Core i7-2720QM CPU 2.20GHz (Sandy Bridge)
Slower 0.78 +0 ubuntu 18.04 X64 Intel Core i7-8700 CPU 3.20GHz (Coffee Lake)
Slower 0.80 +0 ubuntu 20.04 X64 AMD Ryzen 9 5900X
Slower 0.87 +0 ubuntu 20.04 X64 Intel Core i9-10900K CPU 3.70GHz
Same 0.94 +0 Windows 10 X86 Intel Xeon CPU E5-1650 v4 3.60GHz
Same 0.99 +0 Windows 10 X86 Intel Core i7-6700 CPU 3.40GHz (Skylake)
Same 0.96 +0 Windows 11 X86 AMD Ryzen Threadripper PRO 3945WX 12-Cores
Slower 0.85 +0 macOS Big Sur 11.6.8 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)
Slower 0.87 +0 macOS Monterey 12.3.1 X64 Intel Core i7-5557U CPU 3.10GHz (Broadwell)
Slower 0.80 +0 macOS Monterey 12.4 X64 Intel Core i5-4278U CPU 2.60GHz (Haswell)

@mrsharm mrsharm mentioned this issue Aug 12, 2022
Closed
24 tasks
@ghost ghost locked as resolved and limited conversation to collaborators Sep 11, 2022
@jeffhandley jeffhandley added runtime-coreclr specific to the CoreCLR runtime and removed CoreClr labels Dec 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dakersnar dakersnar

Labels
arch-arm64 area-System.Runtime runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
7.0.0
Development

No branches or pull requests
6 participants
@jeffhandley @stephentoub @jeffschwMSFT @tarekgh @dakersnar @mrsharm