Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 31 Regressions on 11/16/2022 6:51:59 PM #78835

Closed
performanceautofiler bot opened this issue Nov 22, 2022 · 6 comments
Closed

[Perf] Linux/x64: 31 Regressions on 11/16/2022 6:51:59 PM #78835

performanceautofiler bot opened this issue Nov 22, 2022 · 6 comments
Assignees
@kg
Labels
arch-x64 area-Codegen-meta-mono os-linux Linux OS (any supported distro)

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Tests.Perf_Double

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToStringWithCultureInfo - Duration of single invocation 1.59 μs 1.73 μs 1.09 0.03 False
ToStringWithFormat - Duration of single invocation 1.65 μs 1.76 μs 1.07 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Double*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Double.ToStringWithCultureInfo(value: 12345, culture: zh)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7335546806939828 > 1.707420676592441.
IsChangePoint: Marked as a change because one of 9/28/2022 6:26:25 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -12.180990931897776 (T) = (0 -1757.2439491613184) / Math.Sqrt((1072.675673308482 / (22)) + (820.0884386234887 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.06678355574728254 = (1647.2356924646892 - 1757.2439491613184) / 1647.2356924646892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Double.ToStringWithFormat(value: -1.7976931348623157E+308, format: "E")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7622719154972777 > 1.7298543453137503.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -16.839692446327312 (T) = (0 -1769.7606005178861) / Math.Sqrt((709.0974623652461 / (19)) + (304.6377998032078 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.07073405477337122 = (1652.8479622257546 - 1769.7606005178861) / 1652.8479622257546 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Uri_IsNotMatch - Duration of single invocation 4.23 μs 4.58 μs 1.08 0.06 False
IP_IsNotMatch - Duration of single invocation 9.78 μs 10.66 μs 1.09 0.03 False
Date_IsMatch - Duration of single invocation 2.62 μs 2.88 μs 1.10 0.05 False
Email_IsNotMatch - Duration of single invocation 8.93 μs 9.75 μs 1.09 0.04 False
MatchesSet - Duration of single invocation 1.63 ms 1.74 ms 1.07 0.02 False
Date_IsMatch - Duration of single invocation 2.67 μs 2.84 μs 1.07 0.04 False
Date_IsMatch - Duration of single invocation 2.69 μs 2.86 μs 1.07 0.02 False
OneNodeBacktracking - Duration of single invocation 15.49 μs 16.86 μs 1.09 0.07 False
Date_IsNotMatch - Duration of single invocation 5.58 μs 5.98 μs 1.07 0.02 False
Email_IsMatch - Duration of single invocation 6.47 μs 6.91 μs 1.07 0.03 False
Backtracking - Duration of single invocation 25.10 μs 26.92 μs 1.07 0.04 False
Uri_IsNotMatch - Duration of single invocation 4.32 μs 4.74 μs 1.10 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.5806447861552035 > 4.534006537736631.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/11/2022 6:23:40 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -12.094648872877148 (T) = (0 -4639.9790574831595) / Math.Sqrt((8636.147626665994 / (22)) + (4558.084324097405 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.0666640700321465 = (4349.990955768598 - 4639.9790574831595) / 4349.990955768598 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.IP_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.6640028774543 > 10.321284735310623.
IsChangePoint: Marked as a change because one of 10/23/2022 1:14:54 PM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -11.952893923082966 (T) = (0 -10509.151068481535) / Math.Sqrt((35836.62376630446 / (22)) + (10867.732504006335 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.054483221984350134 = (9966.16242855451 - 10509.151068481535) / 9966.16242855451 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8768037424637947 > 2.765145445257792.
IsChangePoint: Marked as a change because one of 10/14/2022 1:19:37 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -18.696473110103444 (T) = (0 -2880.704329389745) / Math.Sqrt((1092.313483338753 / (19)) + (2249.605887625718 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.08248189798724273 = (2661.2032355886054 - 2880.704329389745) / 2661.2032355886054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.751399321587687 > 9.38185561654354.
IsChangePoint: Marked as a change because one of 8/5/2022 6:57:51 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -11.385720122910907 (T) = (0 -9520.964352027262) / Math.Sqrt((29781.822921634575 / (22)) + (8782.436671867905 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.051944140072749743 = (9050.826930191195 - 9520.964352027262) / 9050.826930191195 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.MatchesSet(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7373450187500001 > 1.7103555629495195.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/15/2022 10:02:59 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -18.980432851462965 (T) = (0 -1749484.3688405964) / Math.Sqrt((306394503.9211237 / (20)) + (476725100.19948405 / (26))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (26) - 2, .025) and -0.06716687678904465 = (1639372.8168405576 - 1749484.3688405964) / 1639372.8168405576 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8428151408988214 > 2.7888468182763932.
IsChangePoint: Marked as a change because one of 10/14/2022 1:19:37 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -16.341496421582367 (T) = (0 -2864.0799025900988) / Math.Sqrt((1341.4607071091677 / (19)) + (2814.882641849112 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.08066132751077526 = (2650.3029484614744 - 2864.0799025900988) / 2650.3029484614744 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8605259423785996 > 2.789053668516881.
IsChangePoint: Marked as a change because one of 10/14/2022 1:19:37 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -17.000655076077084 (T) = (0 -2861.7185971974) / Math.Sqrt((520.1188553051064 / (19)) + (3339.7216967147856 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.07751846294845163 = (2655.8418213705395 - 2861.7185971974) / 2655.8418213705395 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.OneNodeBacktracking(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.855934743495997 > 16.349400876029073.
IsChangePoint: Marked as a change because one of 8/1/2022 7:13:37 AM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -13.154906500515974 (T) = (0 -16684.415967642297) / Math.Sqrt((52402.042648659524 / (19)) + (113978.01505215642 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.06969540998337106 = (15597.352117180408 - 16684.415967642297) / 15597.352117180408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Date_IsNotMatch(Options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.9837972371450485 > 5.93609853273178.
IsChangePoint: Marked as a change because one of 10/14/2022 11:50:37 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -12.370778709432454 (T) = (0 -6093.247270886613) / Math.Sqrt((9709.475818045436 / (19)) + (13346.664950502558 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.06815424265565943 = (5704.463856958991 - 6093.247270886613) / 5704.463856958991 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.914646241758242 > 6.781077925033604.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -11.24480005682823 (T) = (0 -6847.834617336331) / Math.Sqrt((9346.802004905334 / (19)) + (9993.244190202719 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.05024606779357432 = (6520.219239404257 - 6847.834617336331) / 6520.219239404257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Backtracking(Options: IgnoreCase, Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.922831747565866 > 26.164214092882364.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -14.393533235352736 (T) = (0 -26698.026899685385) / Math.Sqrt((114452.77640565122 / (19)) + (210770.99858462848 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.06696205294861218 = (25022.470879731685 - 26698.026899685385) / 25022.470879731685 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.RegularExpressions.Tests.Perf_Regex_Common.Uri_IsNotMatch(Options: Compiled)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.736952221971294 > 4.568332628819631.
IsChangePoint: Marked as a change because one of 9/28/2022 9:30:44 PM, 10/6/2022 10:14:50 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -11.698231243311364 (T) = (0 -4593.3181935412895) / Math.Sqrt((3340.823969612707 / (19)) + (9224.411447218268 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.06072401115393958 = (4330.361286480462 - 4593.3181935412895) / 4330.361286480462 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.IO.Tests.Perf_FileStream

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Read_NoBuffering - Duration of single invocation 123.95 μs 137.75 μs 1.11 0.08 False
SeekBackward - Duration of single invocation 1.12 ms 1.36 ms 1.21 0.16 False
CopyToFile - Duration of single invocation 62.33 μs 70.10 μs 1.12 0.15 False
SeekForward - Duration of single invocation 278.89 μs 303.64 μs 1.09 0.06 False
SeekForward - Duration of single invocation 275.50 μs 306.72 μs 1.11 0.06 False
WriteByte - Duration of single invocation 89.70 μs 95.74 μs 1.07 0.05 False
LockUnlock - Duration of single invocation 19.48 μs 22.30 μs 1.15 0.04 False
OpenClose - Duration of single invocation 11.87 μs 12.52 μs 1.05 0.05 False
WriteAsync_NoBuffering_PreallocationSize - Duration of single invocation 1.07 ms 1.23 ms 1.14 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_FileStream*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_FileStream.Read_NoBuffering(fileSize: 1048576, userBufferSize: 16384, options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 137.75456880482454 > 129.87798343111112.
IsChangePoint: Marked as a change because one of 10/14/2022 3:33:41 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.958913475058198 (T) = (0 -135013.12462213452) / Math.Sqrt((1931505.0772461717 / (22)) + (40092360.168973155 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.09445191521746091 = (123361.40377195853 - 135013.12462213452) / 123361.40377195853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.SeekBackward(fileSize: 1024, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3553412775974027 > 1.1939230661229399.
IsChangePoint: Marked as a change because one of 11/16/2022 6:51:59 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -13.034256297439198 (T) = (0 -1338942.0912818373) / Math.Sqrt((529944977.4870485 / (23)) + (4437096802.170046 / (24))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (24) - 2, .025) and -0.16329049239718185 = (1150995.4736436398 - 1338942.0912818373) / 1150995.4736436398 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.CopyToFile(fileSize: 1024, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 70.09667525856497 > 65.95235059923094.
IsChangePoint: Marked as a change because one of 10/5/2022 3:48:29 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -5.390012459056491 (T) = (0 -66580.54570748685) / Math.Sqrt((856138.8591699478 / (22)) + (9674864.6013826 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.05577949166355808 = (63062.92765980707 - 66580.54570748685) / 63062.92765980707 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.SeekForward(fileSize: 1024, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.63621033653845 > 286.4054396497845.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -22.194082855348604 (T) = (0 -305309.286926717) / Math.Sqrt((15901030.980619308 / (19)) + (30278937.666148465 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.11195433999245782 = (274569.9854265489 - 305309.286926717) / 274569.9854265489 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.SeekForward(fileSize: 1024, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 306.7204803921569 > 289.8852387612103.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -15.438081547110471 (T) = (0 -307424.1064182726) / Math.Sqrt((38845534.04386012 / (19)) + (59545432.77804823 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.11428178445466218 = (275894.4018534129 - 307424.1064182726) / 275894.4018534129 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.WriteByte(fileSize: 1024, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 95.73775492075666 > 94.85691810044644.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/15/2022 8:48:03 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -6.531176223374755 (T) = (0 -95474.96578404456) / Math.Sqrt((5058213.429189074 / (19)) + (6758893.66622662 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.05123504922259823 = (90821.71095289252 - 95474.96578404456) / 90821.71095289252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.LockUnlock(fileSize: 1024, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.3024334258596 > 20.570126137311853.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/26/2022 4:56:52 PM, 11/16/2022 6:51:59 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.534872883563231 (T) = (0 -21649.031297163976) / Math.Sqrt((64231.8944584602 / (23)) + (963080.1621661055 / (24))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (24) - 2, .025) and -0.08894010891667697 = (19880.828265845892 - 21649.031297163976) / 19880.828265845892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.OpenClose(fileSize: 1024, options: None)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.521625648748842 > 12.416958425161772.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/26/2022 4:56:52 PM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -7.121141126149974 (T) = (0 -12938.32924960464) / Math.Sqrt((49260.38870537652 / (22)) + (394924.4152283816 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.07981646278267197 = (11981.970728861408 - 12938.32924960464) / 11981.970728861408 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_FileStream.WriteAsync_NoBuffering_PreallocationSize(fileSize: 1048576, userBufferSize: 16384, options: Asynchronous)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2258042938988094 > 1.1336110320546875.
IsChangePoint: Marked as a change because one of 9/30/2022 12:37:41 PM, 11/16/2022 6:51:59 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -9.846607450035139 (T) = (0 -1200003.4294687638) / Math.Sqrt((351870973.96769196 / (23)) + (3038643627.3369856 / (24))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (24) - 2, .025) and -0.10833809595218238 = (1082705.2086825825 - 1200003.4294687638) / 1082705.2086825825 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Tests.Perf_Array

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ArrayCreate2D - Duration of single invocation 7.35 μs 8.60 μs 1.17 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Array*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Array.ArrayCreate2D

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.595802453228996 > 7.433317899579019.
IsChangePoint: Marked as a change because one of 10/3/2022 1:11:02 PM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -5.142253099358792 (T) = (0 -7699.820682438782) / Math.Sqrt((48989.69199352059 / (22)) + (378991.56163951085 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.09656332703779327 = (7021.774750792305 - 7699.820682438782) / 7021.774750792305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Buffers.Tests.RentReturnArrayPoolTests<Object>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ProducerConsumer - Duration of single invocation 4.08 μs 4.54 μs 1.11 0.08 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.RentReturnArrayPoolTests&lt;Object&gt;*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Tests.RentReturnArrayPoolTests<Object>.ProducerConsumer(RentalSize: 4096, ManipulateArray: False, Async: False, UseSharedPool: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.537051057368421 > 4.273867608365385.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.526319343994576 (T) = (0 -4446.976861612858) / Math.Sqrt((8535.745117024639 / (22)) + (31082.906886259316 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.08394011428214813 = (4102.603827479823 - 4446.976861612858) / 4102.603827479823 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.IO.Tests.Perf_RandomAccess

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadAsync - Duration of single invocation 2.12 ms 2.49 ms 1.18 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_RandomAccess*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_RandomAccess.ReadAsync(fileSize: 1048576, bufferSize: 4096, options: None)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4938779549107144 > 2.259295968299279.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -6.056063401200131 (T) = (0 -2363097.2531185653) / Math.Sqrt((2764050014.3203826 / (22)) + (21084785693.69904 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.088388703085571 = (2171188.6997900736 - 2363097.2531185653) / 2171188.6997900736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Net.Http.Tests.SocketsHttpHandlerPerfTest

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Get - Duration of single invocation 212.79 μs 249.85 μs 1.17 0.33 False
Get_EnumerateHeaders_Unvalidated - Duration of single invocation 181.07 μs 211.83 μs 1.17 0.13 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Http.Tests.SocketsHttpHandlerPerfTest*'

Payloads

Baseline
Compare

Histogram

System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get(ssl: True, chunkedResponse: True, responseLength: 1)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 249.8461537909836 > 225.1583011352993.
IsChangePoint: Marked as a change because one of 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -7.326593958769699 (T) = (0 -235668.09117334962) / Math.Sqrt((39804438.328785874 / (22)) + (285192425.607333 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.12742520947919717 = (209032.12842137364 - 235668.09117334962) / 209032.12842137364 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Net.Http.Tests.SocketsHttpHandlerPerfTest.Get_EnumerateHeaders_Unvalidated(ssl: False, chunkedResponse: False, responseLength: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 211.828082322801 > 190.4254554570175.
IsChangePoint: Marked as a change because one of 10/4/2022 3:13:24 PM, 11/16/2022 6:51:59 PM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -6.584661504222718 (T) = (0 -202129.27147174728) / Math.Sqrt((10607218.212371217 / (23)) + (258722545.63395128 / (24))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (24) - 2, .025) and -0.12261485658234238 = (180052.1971418622 - 202129.27147174728) / 180052.1971418622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.IO.Tests.BinaryWriterExtendedTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteAsciiString - Duration of single invocation 14.49 ms 15.43 ms 1.06 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.BinaryWriterExtendedTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.BinaryWriterExtendedTests.WriteAsciiString(StringLengthInChars: 2000000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.434487182692308 > 15.218713079609376.
IsChangePoint: Marked as a change because one of 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -4.5102914878441736 (T) = (0 -15590061.632980311) / Math.Sqrt((83314963537.15659 / (22)) + (1011686552553.2931 / (25))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (25) - 2, .025) and -0.06480453809270928 = (14641242.66497344 - 15590061.632980311) / 14641242.66497344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryParse - Duration of single invocation 161.54 ns 170.85 ns 1.06 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryParse(value: "12345")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 170.85070113767176 > 168.93661026967817.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -6.524192211107622 (T) = (0 -174.05686925489294) / Math.Sqrt((27.38576159684356 / (22)) + (32.134666070798104 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.06411341201741394 = (163.56984818460737 - 174.05686925489294) / 163.56984818460737 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 715fde2b08357dfb72248f18ab09dd27cf7f2888
Compare e2c0735927642a11162c0d12799be2921bd1c984
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Exists - Duration of single invocation 3.43 μs 3.86 μs 1.12 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_File.Exists

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.8556128720787215 > 3.593933831273345.
IsChangePoint: Marked as a change because one of 8/5/2022 10:47:59 PM, 9/28/2022 6:40:13 PM, 11/16/2022 11:42:33 AM, 11/22/2022 2:20:42 AM falls between 11/9/2022 12:33:05 PM and 11/22/2022 2:20:42 AM.
IsRegressionStdDev: Marked as regression because -8.22633114476071 (T) = (0 -3697.94536607219) / Math.Sqrt((1219.0800202264245 / (22)) + (23048.1722963717 / (24))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (24) - 2, .025) and -0.0763091919396377 = (3435.764921237968 - 3697.94536607219) / 3435.764921237968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added refs/heads/main untriaged New issue has not been triaged by the area owner labels Nov 22, 2022
@kotlarmilos
Copy link
Member

Looks like regression in ce58915. @kg do you have any idea what caused regressions?

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 23, 2022
@kotlarmilos kotlarmilos mentioned this issue Nov 23, 2022
Closed
@kg
Copy link
Member

kg commented Nov 23, 2022

Why are the diff links in these reports nonsense? Is there a way to get a meaningful diff?
image

@kotlarmilos
Copy link
Member

kotlarmilos commented Nov 24, 2022 •
edited
Loading

Strange, could be an issue. For which microbenchmark have you encountered this?

Here is the range: 3cfd6cf...ce58915

@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Nov 24, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@kotlarmilos
Copy link
Member

according to the latest data it looks like nose

@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@kg kg

Labels
arch-x64 area-Codegen-meta-mono os-linux Linux OS (any supported distro)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kg @jeffhandley @jeffschwMSFT @kotlarmilos