Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 13 Regressions on 9/14/2022 12:06:24 AM #8642

Closed
performanceautofiler bot opened this issue Sep 20, 2022 · 4 comments
Closed

[Perf] Linux/arm64: 13 Regressions on 9/14/2022 12:06:24 AM #8642

performanceautofiler bot opened this issue Sep 20, 2022 · 4 comments
Labels
arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Sep 20, 2022 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline bf362e876cba86a19fd6a721a4e694526361e661
Compare 9028dc2d715a827a21a5ffbd029bc7a24073406b
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Count - Duration of single invocation 12.60 ms 17.45 ms 1.38 0.42 False
Count - Duration of single invocation 18.93 ms 41.66 ms 2.20 0.39 False
Count - Duration of single invocation 39.41 ms 57.37 ms 1.46 0.45 False
Count - Duration of single invocation 17.18 ms 64.40 ms 3.75 0.33 False
Count - Duration of single invocation 16.90 ms 63.80 ms 3.78 0.40 False
Count - Duration of single invocation 58.47 ms 138.85 ms 2.37 0.39 False
Count - Duration of single invocation 51.85 ms 132.64 ms 2.56 0.40 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "[a-q][^u-z]{13}x", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -2.8263447754012128 (T) = (0 -41016866.31892857) / Math.Sqrt((4842648989382.501 / (4)) + (462732793236905.9 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -2.0049323522468647 = (13649846.822095416 - 41016866.31892857) / 13649846.822095416 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sher[a-z]+|Hol[a-z]+", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -2.406483858878667 (T) = (0 -66681621.70607142) / Math.Sqrt((192510416829965.28 / (4)) + (1191261419519346.2 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -1.5688999345368002 = (25957267.081364468 - 66681621.70607142) / 25957267.081364468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Holmes", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.177096588284466 (T) = (0 -101980059.10114035) / Math.Sqrt((49736541919928.31 / (4)) + (1641775660827533.5 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -1.3536031043361092 = (43329335.73772894 - 101980059.10114035) / 43329335.73772894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/18/2021 11:57:31 PM, 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -458.78312802765237 (T) = (0 -64136499.23330281) / Math.Sqrt((439240442.30453336 / (3)) + (62017039049.73369 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -2.7366401329566266 = (17164216.234693874 - 64136499.23330281) / 17164216.234693874 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock Holmes", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -2.821346388580035 (T) = (0 -74998464.64354396) / Math.Sqrt((409790109441632.1 / (3)) + (802494637578380.6 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -1.6214683564505148 = (28609334.32936508 - 74998464.64354396) / 28609334.32936508 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "Sherlock|Holmes|Watson", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.375221708341805 (T) = (0 -102601058.99464285) / Math.Sqrt((465022909148582.3 / (3)) + (1579767275203673.5 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -2.056372839161493 = (33569549.395286195 - 102601058.99464285) / 33569549.395286195 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock", Options: NonBacktracking)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.5582561032088083 (T) = (0 -97801262.74679488) / Math.Sqrt((406380135711890.5 / (3)) + (1458606681438182.5 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -2.423272202775354 = (28569525.574829932 - 97801262.74679488) / 28569525.574829932 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 18.04
Baseline bf362e876cba86a19fd6a721a4e694526361e661
Compare 9028dc2d715a827a21a5ffbd029bc7a24073406b
Diff Diff

Regressions in System.IO.Tests.StringReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLine - Duration of single invocation 8.48 ms 9.77 ms 1.15 0.55 False
ReadLineAsync - Duration of single invocation 335.58 μs 2.61 ms 7.79 0.42 False
ReadLine - Duration of single invocation 326.83 μs 944.17 μs 2.89 0.40 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.IO.Tests.StringReaderReadLineTests*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [ 0, 0])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -2.7317590372474267 (T) = (0 -14083520.046225747) / Math.Sqrt((213602729955.4725 / (4)) + (17676736958404.97 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -0.5809120138993132 = (8908478.095177986 - 14083520.046225747) / 8908478.095177986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.403415456770035 (T) = (0 -1715473.9162510897) / Math.Sqrt((4751018.601065367 / (3)) + (983479034320.7555 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -4.082073279115223 = (337553.9513176303 - 1715473.9162510897) / 337553.9513176303 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StringReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 8/4/2022 11:58:08 PM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -2.4787564272834293 (T) = (0 -1103115.0634254918) / Math.Sqrt((8653293.698973745 / (3)) + (585054621769.805 / (6))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (6) - 2, .025) and -2.352158600972108 = (329076.0356940135 - 1103115.0634254918) / 329076.0356940135 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline bf362e876cba86a19fd6a721a4e694526361e661
Compare 9028dc2d715a827a21a5ffbd029bc7a24073406b
Diff Diff

Regressions in System.Text.Json.Tests.Utf8JsonReaderCommentsTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Utf8JsonReaderCommentParsing - Duration of single invocation 35.60 μs 119.71 μs 3.36 0.38 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.Json.Tests.Utf8JsonReaderCommentsTests*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Text.Json.Tests.Utf8JsonReaderCommentsTests.Utf8JsonReaderCommentParsing(CommentHandling: Allow, SegmentSize: 0, TestCase: LongSingleLine)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.246989539567821 (T) = (0 -284614.1245641567) / Math.Sqrt((1661129299.927316 / (4)) + (22700719333.15632 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -4.0784969755823015 = (56042.983964073894 - 284614.1245641567) / 56042.983964073894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline bf362e876cba86a19fd6a721a4e694526361e661
Compare 9028dc2d715a827a21a5ffbd029bc7a24073406b
Diff Diff

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IndexOfString - Duration of single invocation 9.01 μs 12.75 μs 1.42 0.33 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Memory.ReadOnlySpan*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "X", comparisonType: Ordinal)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/12/2022 1:22:26 AM, 3/18/2022 1:59:24 AM, 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -3.5121034962167803 (T) = (0 -12740.840879763839) / Math.Sqrt((2742881.39977672 / (4)) + (219.4130779869954 / (5))) is less than -2.3646242515927898 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (5) - 2, .025) and -0.2957968531387057 = (9832.43696641392 - 12740.840879763839) / 9832.43696641392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 18.04
Baseline bf362e876cba86a19fd6a721a4e694526361e661
Compare 9028dc2d715a827a21a5ffbd029bc7a24073406b
Diff Diff

Regressions in System.Tests.Perf_GC<Byte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
NewOperator_Array - Duration of single invocation 335.31 ns 370.78 ns 1.11 0.36 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Tests.Perf_GC&lt;Byte&gt;*'

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Tests.Perf_GC<Byte>.NewOperator_Array(length: 1000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 9/13/2022 1:36:49 AM, 9/20/2022 1:47:12 AM falls between 8/4/2022 11:58:08 PM and 9/20/2022 1:47:12 AM.
IsRegressionStdDev: Marked as regression because -5.09285676866701 (T) = (0 -373.1758631439536) / Math.Sqrt((210.41494268938467 / (4)) + (11.467462877845959 / (2))) is less than -2.7764451051977974 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (4) + (2) - 2, .025) and -0.1163666568576876 = (334.27714886644577 - 373.1758631439536) / 334.27714886644577 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Sep 20, 2022
@naricc naricc removed the untriaged label Sep 28, 2022
@naricc
Copy link

naricc commented Sep 28, 2022

Also missing data

@kotlarmilos kotlarmilos mentioned this issue Oct 26, 2022
Closed
@kotlarmilos
Copy link
Member

Hey @stephentoub do you think that the following changes could have caused the regressions?

dotnet/runtime@b1a309d
dotnet/runtime@a037f0c

@stephentoub
Copy link
Member

Hey @stephentoub do you think that the following changes could have caused the regressions?

dotnet/runtime@b1a309d dotnet/runtime@a037f0c

The former only touched code in string.Split, the latter in string.Substring.

Most of these tests don't use either, directly or indirectly, so these commits should not impact them.

The only one that does is StringReader, where its ReadLine{Async} will use string.Substring. And there's no way that change to Substring would cause an 8x regression in ReadLine; or, rather, if it did, something is very wrong in the mono interpreter.

@kotlarmilos
Copy link
Member

Fixed by dotnet/runtime@254844a

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@stephentoub @kotlarmilos @naricc