Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 22 Regressions on 1/5/2023 2:25:12 AM #11449

Closed
performanceautofiler bot opened this issue Jan 10, 2023 · 2 comments
Closed

[Perf] Linux/x64: 22 Regressions on 1/5/2023 2:25:12 AM #11449

performanceautofiler bot opened this issue Jan 10, 2023 · 2 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 10, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2072f16cc1b3327d901091772189650de93fe49b
Compare 83ce1253cf71348776fec4464e3fad1277fac7f0
Diff Diff

Regressions in System.Text.Json.Node.Tests.Perf_ParseThenWrite

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ParseThenWrite - Duration of single invocation 11.85 μs 24.81 μs 2.09 0.01 False
ParseThenWrite - Duration of single invocation 140.59 μs 799.73 μs 5.69 0.01 False
ParseThenWrite - Duration of single invocation 176.91 μs 866.16 μs 4.90 0.01 False
ParseThenWrite - Duration of single invocation 289.35 μs 1.16 ms 4.02 0.01 False
ParseThenWrite - Duration of single invocation 15.66 ms 81.51 ms 5.20 0.02 False
ParseThenWrite - Duration of single invocation 54.22 μs 216.73 μs 4.00 0.02 False
ParseThenWrite - Duration of single invocation 211.75 μs 692.59 μs 3.27 0.01 False
ParseThenWrite - Duration of single invocation 33.82 μs 135.94 μs 4.02 0.01 False
ParseThenWrite - Duration of single invocation 12.40 ms 75.55 ms 6.09 0.02 False
ParseThenWrite - Duration of single invocation 37.56 μs 132.05 μs 3.52 0.02 False
ParseThenWrite - Duration of single invocation 225.33 μs 1.02 ms 4.54 0.01 False
ParseThenWrite - Duration of single invocation 155.76 μs 602.35 μs 3.87 0.01 False
ParseThenWrite - Duration of single invocation 29.38 μs 125.42 μs 4.27 0.01 False
ParseThenWrite - Duration of single invocation 10.86 μs 23.83 μs 2.19 0.01 False
ParseThenWrite - Duration of single invocation 42.39 μs 140.51 μs 3.31 0.02 False
ParseThenWrite - Duration of single invocation 65.78 μs 234.72 μs 3.57 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Node.Tests.Perf_ParseThenWrite*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: HelloWorld)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.807646172288358 > 12.476900909034471.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -284.02530964727515 (T) = (0 -24817.938916529376) / Math.Sqrt((19318.319922633564 / (24)) + (29997.521573059556 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -1.107881564566727 = (11773.877305877324 - 24817.938916529376) / 11773.877305877324 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 799.7295001201923 > 148.05413261292617.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -670.9267134642828 (T) = (0 -798328.3468518982) / Math.Sqrt((1103985.5572627382 / (24)) + (20108481.95355749 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -4.663919231810359 = (140949.81128407238 - 798328.3468518982) / 140949.81128407238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 866.1593551655955 > 183.92660780021063.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -553.4432724761074 (T) = (0 -872828.3263840964) / Math.Sqrt((4225789.042443596 / (24)) + (32533232.73887949 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -3.9921721628414164 = (174839.38812865477 - 872828.3263840964) / 174839.38812865477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1632582296806968 > 303.8709537195362.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -509.05801810290365 (T) = (0 -1171010.2061766041) / Math.Sqrt((22914487.611166667 / (24)) + (44796379.91784001 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -3.0292973665083665 = (290623.9226496595 - 1171010.2061766041) / 290623.9226496595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 81.50750294642857 > 16.526681027791668.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -466.898027005513 (T) = (0 -82330585.56926823) / Math.Sqrt((56401280367.45006 / (24)) + (397073292811.4927 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -4.2620496781986335 = (15646105.720053293 - 82330585.56926823) / 15646105.720053293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.7300801215278 > 57.716958829400276.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -361.360696697042 (T) = (0 -215059.36316276804) / Math.Sqrt((1089941.274600631 / (24)) + (3360124.2835925384 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -2.9675724730714483 = (54204.268383857 - 215059.36316276804) / 54204.268383857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 692.5917876552796 > 223.22407019214526.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -584.2674094250474 (T) = (0 -686095.7997335092) / Math.Sqrt((4672688.868879119 / (24)) + (10119740.498152988 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -2.2158179905176643 = (213350.32074469657 - 686095.7997335092) / 213350.32074469657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 135.94047414596275 > 35.55429041865451.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -602.0446764338474 (T) = (0 -136370.08717308496) / Math.Sqrt((46182.88565011884 / (24)) + (623512.104700177 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -3.036247049489846 = (33786.3578470305 - 136370.08717308496) / 33786.3578470305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 75.55201959615384 > 13.019826621052633.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -386.49464863377324 (T) = (0 -75166709.10676199) / Math.Sqrt((17774796676.9589 / (24)) + (564381483656.5588 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -5.07382363505748 = (12375517.239734711 - 75166709.10676199) / 12375517.239734711 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.05423531162467 > 38.94056277788077.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -484.8787805049631 (T) = (0 -131856.90613285697) / Math.Sqrt((131770.833913612 / (24)) + (747690.1876914226 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -2.5314189496718473 = (37338.22240068956 - 131856.90613285697) / 37338.22240068956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0219995114583333 > 235.8445407064471.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -459.59473655106194 (T) = (0 -1030067.8762560885) / Math.Sqrt((9971966.372304033 / (24)) + (61434971.99613585 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -3.6270085349584664 = (222620.69941596396 - 1030067.8762560885) / 222620.69941596396 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 602.3506241414833 > 165.49846372632572.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -448.06762233008106 (T) = (0 -597600.5834337625) / Math.Sqrt((2740976.1326645543 / (24)) + (18859640.457984865 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -2.831452727590025 = (155972.323273228 - 597600.5834337625) / 155972.323273228 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 125.42087291666668 > 30.95648132661421.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -514.5625179351032 (T) = (0 -127397.88830761806) / Math.Sqrt((50629.55379053436 / (24)) + (752193.5632209115 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -3.3391183087944207 = (29360.316829668158 - 127397.88830761806) / 29360.316829668158 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.828524604689207 > 11.519776407150374.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -252.3201543638852 (T) = (0 -23547.74726301173) / Math.Sqrt((20354.23011524868 / (24)) + (36487.73222655124 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -1.1573687228130591 = (10915.0313592695 - 23547.74726301173) / 10915.0313592695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 140.5080214285714 > 44.67537113441009.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -302.8790346174585 (T) = (0 -143570.19690491402) / Math.Sqrt((310427.05849787005 / (24)) + (2150935.1087519196 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -2.3547935952408796 = (42795.53803506216 - 143570.19690491402) / 42795.53803506216 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Node.Tests.Perf_ParseThenWrite.ParseThenWrite(IsDataIndented: True, TestCase: LotsOfNumbers)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 234.7188518943743 > 68.6868047858107.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -345.90655152502273 (T) = (0 -236457.71176318164) / Math.Sqrt((1256561.4054658327 / (24)) + (4451779.542634856 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -2.642057827997082 = (64924.20574585426 - 236457.71176318164) / 64924.20574585426 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 2072f16cc1b3327d901091772189650de93fe49b
Compare 83ce1253cf71348776fec4464e3fad1277fac7f0
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateArray - Duration of single invocation 0.00 ns 35.98 μs ∞ 0.12 False
EnumerateArray - Duration of single invocation 0.00 ns 37.28 μs ∞ 0.14 False
EnumerateArray - Duration of single invocation 0.00 ns 34.48 μs ∞ 0.10 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfNumbers)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.977539061196396 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -194.3554766657398 (T) = (0 -34379.17990077524) / Math.Sqrt((0 / (24)) + (719656.5711704377 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -∞ = (0 - 34379.17990077524) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 37.283269504841954 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -209.11958922326784 (T) = (0 -36841.030410955405) / Math.Sqrt((0 / (24)) + (713841.7403274592 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -∞ = (0 - 36841.030410955405) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateArray(TestCase: ArrayOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.47926012101861 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -275.8367628076235 (T) = (0 -34104.511147472804) / Math.Sqrt((0 / (24)) + (336311.7521458248 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -∞ = (0 - 34104.511147472804) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2072f16cc1b3327d901091772189650de93fe49b
Compare 83ce1253cf71348776fec4464e3fad1277fac7f0
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateProperties - Duration of single invocation 0.00 ns 51.53 μs ∞ 0.14 False
EnumerateProperties - Duration of single invocation 0.00 ns 48.64 μs ∞ 0.14 False
EnumerateProperties - Duration of single invocation 0.00 ns 48.50 μs ∞ 0.14 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: ObjectProperties)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 51.532595989461356 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -168.42026612640112 (T) = (0 -52655.9283963955) / Math.Sqrt((0 / (24)) + (2150445.9486569925 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -∞ = (0 - 52655.9283963955) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: NumericProperties)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.635345920973556 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -260.7164170394254 (T) = (0 -49553.27658073346) / Math.Sqrt((0 / (24)) + (830875.2654585448 / (23))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (23) - 2, .025) and -∞ = (0 - 49553.27658073346) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateObject.EnumerateProperties(TestCase: StringProperties)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 48.50391520899894 > 0.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/4/2023 8:35:43 PM, 1/10/2023 1:28:02 AM falls between 1/1/2023 3:57:25 AM and 1/10/2023 1:28:02 AM.
IsRegressionStdDev: Marked as regression because -196.0960518459781 (T) = (0 -49620.81455531727) / Math.Sqrt((0 / (24)) + (1408681.3331699744 / (22))) is less than -2.0153675744421933 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (22) - 2, .025) and -∞ = (0 - 49620.81455531727) / 0 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added LLVM=false PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Jan 10, 2023
This was referenced Jan 11, 2023
Closed
Closed
@kotlarmilos
Copy link
Member

@BrzVlad could it be dotnet/runtime@5625fee? It seems like only relevant change in dotnet/runtime@2975fa4...a224371.

@kotlarmilos
Copy link
Member

There was a transient improvement. Not a regression.

newplot (2)

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos