Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 14 Regressions on 11/22/2022 2:57:03 PM #10158

Closed
performanceautofiler bot opened this issue Nov 29, 2022 · 0 comments
Closed

[Perf] Linux/x64: 14 Regressions on 11/22/2022 2:57:03 PM #10158

performanceautofiler bot opened this issue Nov 29, 2022 · 0 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Nov 29, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline dd0ec9471dde6b6740ae6d9b504b64dec190b438
Compare 47158cc4405595ce66d24c29e71e7ac36b532a9c
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LessThanAnyBenchmark - Duration of single invocation 135.46 ns 147.92 ns 1.09 0.26 False
MultiplyBenchmark - Duration of single invocation 204.95 ns 222.88 ns 1.09 0.14 False
AddOperatorBenchmark - Duration of single invocation 205.11 ns 218.35 ns 1.06 0.14 False
EqualsAnyBenchmark - Duration of single invocation 135.73 ns 146.65 ns 1.08 0.18 False
GreaterThanOrEqualAllBenchmark - Duration of single invocation 136.25 ns 151.38 ns 1.11 0.12 False
GreaterThanAllBenchmark - Duration of single invocation 135.18 ns 145.27 ns 1.07 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;*'

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<Int16>.LessThanAnyBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 147.92150160339745 > 142.4583275927841.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -7.780730629407865 (T) = (0 -147.1681119615814) / Math.Sqrt((13.412408127665207 / (25)) + (32.36222252947122 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.07385678883486514 = (137.04631147441813 - 147.1681119615814) / 137.04631147441813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.MultiplyBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 222.87563795188004 > 215.62421369036517.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -16.8521882705278 (T) = (0 -220.13333365588286) / Math.Sqrt((10.350913807807501 / (25)) + (11.790121027029251 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.07522129202564527 = (204.7330491764782 - 220.13333365588286) / 204.7330491764782 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 218.35155623062056 > 215.93646751482726.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -8.969751500124165 (T) = (0 -218.0716903616685) / Math.Sqrt((32.182239186084026 / (25)) + (16.08023688106134 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.059457399238438756 = (205.83337330828329 - 218.0716903616685) / 205.83337330828329 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.EqualsAnyBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 146.65190630538197 > 142.9001025174549.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -10.118593145968067 (T) = (0 -147.83918950370258) / Math.Sqrt((1.8019827376434112 / (25)) + (33.43668268080922 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.08344421912189653 = (136.4529773609595 - 147.83918950370258) / 136.4529773609595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.GreaterThanOrEqualAllBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 151.3831404366435 > 143.4930068950536.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -15.65257321066002 (T) = (0 -150.82538491875) / Math.Sqrt((8.234424650066225 / (25)) + (10.28479993604877 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.09483775169266448 = (137.76048979455422 - 150.82538491875) / 137.76048979455422 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;Int16&gt;.GreaterThanAllBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 145.27180162101806 > 141.88959258022794.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -8.604445752526516 (T) = (0 -145.02138218569743) / Math.Sqrt((8.56173681332879 / (25)) + (18.311539523871275 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.06295560511188686 = (136.4322098573745 - 145.02138218569743) / 136.4322098573745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline dd0ec9471dde6b6740ae6d9b504b64dec190b438
Compare 47158cc4405595ce66d24c29e71e7ac36b532a9c
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
LessThanBenchmark - Duration of single invocation 154.17 ns 163.41 ns 1.06 0.18 False
MinBenchmark - Duration of single invocation 181.76 ns 193.06 ns 1.06 0.14 False
MaxBenchmark - Duration of single invocation 179.28 ns 195.22 ns 1.09 0.11 False
LessThanOrEqualBenchmark - Duration of single invocation 153.05 ns 165.01 ns 1.08 0.16 False
GreaterThanOrEqualBenchmark - Duration of single invocation 162.01 ns 171.80 ns 1.06 0.15 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_VectorOf<Int16>.LessThanBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 163.40933258669324 > 161.0649805719877.
IsChangePoint: Marked as a change because one of 9/29/2022 8:01:59 PM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -7.708190309879715 (T) = (0 -164.76193892892957) / Math.Sqrt((6.475180063904193 / (25)) + (53.47215936909825 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.07399461217936391 = (153.4103961607335 - 164.76193892892957) / 153.4103961607335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.MinBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 193.0610288727502 > 190.79304965160262.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -8.946258911868167 (T) = (0 -193.87840878569074) / Math.Sqrt((13.505041804348558 / (25)) + (37.33605658642302 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.06742012958810828 = (181.6327080701614 - 193.87840878569074) / 181.6327080701614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.MaxBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.21640910425367 > 188.10501787871536.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -9.378705533266972 (T) = (0 -191.00250996604368) / Math.Sqrt((6.6103799942750685 / (25)) + (36.54830015426228 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.06555239669613855 = (179.2521048784347 - 191.00250996604368) / 179.2521048784347 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.LessThanOrEqualBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 165.00607399400957 > 160.9414733480662.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/21/2022 12:46:21 PM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -7.04470356320755 (T) = (0 -164.49871904890006) / Math.Sqrt((19.039428968843584 / (19)) + (24.49234750607163 / (34))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (34) - 2, .025) and -0.059551705431375695 = (155.25313036226734 - 164.49871904890006) / 155.25313036226734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.GreaterThanOrEqualBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.79845446049444 > 169.88785532973134.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -6.749435314299839 (T) = (0 -173.4946881739103) / Math.Sqrt((23.176047778416144 / (25)) + (63.98730546839717 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.07495113594081132 = (161.397743928207 - 173.4946881739103) / 161.397743928207 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline dd0ec9471dde6b6740ae6d9b504b64dec190b438
Compare 47158cc4405595ce66d24c29e71e7ac36b532a9c
Diff Diff

Regressions in System.Numerics.Tests.Perf_Matrix4x4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CreateFromScalars - Duration of single invocation 38.61 ns 43.55 ns 1.13 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Matrix4x4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Matrix4x4.CreateFromScalars

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 43.54849323795535 > 40.730054738115605.
IsChangePoint: Marked as a change because one of 10/17/2022 3:02:22 AM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -7.308111871874573 (T) = (0 -43.27348682333935) / Math.Sqrt((0.8502844670743652 / (25)) + (7.4682780491933745 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.10206657560589978 = (39.26576468354303 - 43.27348682333935) / 39.26576468354303 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline dd0ec9471dde6b6740ae6d9b504b64dec190b438
Compare 47158cc4405595ce66d24c29e71e7ac36b532a9c
Diff Diff

Regressions in System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SubtractionOperatorBenchmark - Duration of single invocation 207.04 ns 218.90 ns 1.06 0.14 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Runtime.Intrinsics.Tests.Perf_Vector128Of&lt;UInt16&gt;*'

Payloads

Baseline
Compare

Histogram

System.Runtime.Intrinsics.Tests.Perf_Vector128Of<UInt16>.SubtractionOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 218.90428449115814 > 217.76826170270996.
IsChangePoint: Marked as a change because one of 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -11.882948120257408 (T) = (0 -220.5698669250581) / Math.Sqrt((13.561549190310629 / (25)) + (18.64869239219303 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.06295247422505032 = (207.50680042008975 - 220.5698669250581) / 207.50680042008975 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline dd0ec9471dde6b6740ae6d9b504b64dec190b438
Compare 47158cc4405595ce66d24c29e71e7ac36b532a9c
Diff Diff

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ConcurrentDictionary - Duration of single invocation 68.58 μs 74.67 μs 1.09 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 74.66862680003645 > 72.64153617466518.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 10/14/2022 1:19:37 PM, 10/22/2022 11:54:51 PM, 11/22/2022 11:24:47 AM, 11/29/2022 1:43:29 AM falls between 11/20/2022 12:28:28 PM and 11/29/2022 1:43:29 AM.
IsRegressionStdDev: Marked as regression because -11.53334774672798 (T) = (0 -73927.89453980424) / Math.Sqrt((1140368.7694066663 / (25)) + (3606822.5870922115 / (28))) is less than -2.007583770314729 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (28) - 2, .025) and -0.06969771021785183 = (69111.01504064011 - 73927.89453980424) / 69111.01504064011 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos