Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 178 Regressions on 12/21/2022 11:29:15 AM #11147

Closed
performanceautofiler bot opened this issue Dec 27, 2022 · 12 comments
Closed

[Perf] Linux/x64: 178 Regressions on 12/21/2022 11:29:15 AM #11147

performanceautofiler bot opened this issue Dec 27, 2022 · 12 comments
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Tests.StreamReaderReadToEndTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadToEnd - Duration of single invocation 136.33 ms 336.18 ms 2.47 0.14 False
ReadToEndAsync - Duration of single invocation 180.33 ms 392.46 ms 2.18 0.07 False
ReadToEndAsync - Duration of single invocation 179.76 ms 382.04 ms 2.13 0.05 False
ReadToEndAsync - Duration of single invocation 182.05 ms 372.44 ms 2.05 0.06 False
ReadToEnd - Duration of single invocation 143.71 ms 347.47 ms 2.42 0.12 False
ReadToEnd - Duration of single invocation 136.13 ms 343.80 ms 2.53 0.10 False
ReadToEndAsync - Duration of single invocation 179.58 ms 390.73 ms 2.18 0.06 False
ReadToEndAsync - Duration of single invocation 179.01 ms 395.15 ms 2.21 0.07 False
ReadToEnd - Duration of single invocation 145.78 ms 344.54 ms 2.36 0.09 False
ReadToEndAsync - Duration of single invocation 178.33 ms 385.10 ms 2.16 0.04 False
ReadToEnd - Duration of single invocation 146.67 ms 342.94 ms 2.34 0.11 False
ReadToEnd - Duration of single invocation 144.30 ms 345.21 ms 2.39 0.12 False
ReadToEnd - Duration of single invocation 145.07 ms 342.13 ms 2.36 0.11 False
ReadToEnd - Duration of single invocation 144.94 ms 343.89 ms 2.37 0.11 False
ReadToEndAsync - Duration of single invocation 179.38 ms 370.80 ms 2.07 0.06 False
ReadToEndAsync - Duration of single invocation 179.12 ms 381.34 ms 2.13 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.StreamReaderReadToEndTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 1, 1])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 336.18067035714284 > 153.0212814548077.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -147.1219082957207 (T) = (0 -343171245.19132066) / Math.Sqrt((39773280233321.04 / (32)) + (12266657098955.982 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4674393716107093 = (139079909.78003377 - 343171245.19132066) / 139079909.78003377 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 392.45919119999996 > 189.02921950673078.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -113.48308682248454 (T) = (0 -382378389.39805657) / Math.Sqrt((17924512666233.035 / (32)) + (50612134926223.09 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.1976830052679703 = (173991603.19366986 - 382378389.39805657) / 173991603.19366986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 382.0354563333333 > 187.98424393.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -103.01685436056755 (T) = (0 -384941427.80331707) / Math.Sqrt((25493700365353.41 / (32)) + (62099259583778.63 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.2295955978606308 = (172650783.9236321 - 384941427.80331707) / 172650783.9236321 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 372.4448178 > 187.67873432000002.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -127.08346943787078 (T) = (0 -382163781.6576923) / Math.Sqrt((17470418987685.174 / (32)) + (38710771801021.25 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.2029546434711482 = (173477825.69664034 - 382163781.6576923) / 173477825.69664034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 347.46630864285714 > 153.43183152500004.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -219.03494248242717 (T) = (0 -343913157.8180708) / Math.Sqrt((23186089889681.05 / (32)) + (2456293911114.9727 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4448680674866434 = (140667368.67794183 - 343913157.8180708) / 140667368.67794183 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   1,    8])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 343.8031709285714 > 143.76965743500003.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -138.3604386673101 (T) = (0 -344532403.6922568) / Math.Sqrt((29859317259310.92 / (31)) + (22043331906987.965 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -1.4630751684741181 = (139878964.3540175 - 344532403.6922568) / 139878964.3540175 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   1,    8])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 390.73143046666667 > 188.010222015.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -141.15717163894624 (T) = (0 -384401399.96857953) / Math.Sqrt((28580440051193.5 / (32)) + (24301762966747.71 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.2221774945025334 = (172984111.7190476 - 384401399.96857953) / 172984111.7190476 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   1,    1])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 395.14965126666664 > 188.85940006750002.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -127.4481375917007 (T) = (0 -382915895.6874034) / Math.Sqrt((26197703736719.62 / (32)) + (33698552707804.266 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.2025198720164052 = (173853548.6342033 - 382915895.6874034) / 173853548.6342033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 344.5441004 > 148.237089.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -147.32688904825727 (T) = (0 -344928580.7639601) / Math.Sqrt((29719697731571.04 / (32)) + (18450872873838.812 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4815363663179135 = (138997995.5344208 - 344928580.7639601) / 138997995.5344208 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 385.1027226666667 > 187.95916195875003.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -161.9770108359866 (T) = (0 -381647354.5140619) / Math.Sqrt((25949087342242.637 / (32)) + (15034011462600.69 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.1955440729234632 = (173828145.4791667 - 381647354.5140619) / 173828145.4791667 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 342.9384239166667 > 152.91864294.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -172.87687275436045 (T) = (0 -343748442.914764) / Math.Sqrt((23949074257403.41 / (32)) + (11368563552645.027 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.443748054441357 = (140664436.4544958 - 343748442.914764) / 140664436.4544958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 345.21151984615386 > 152.6529010825.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -151.1941908366584 (T) = (0 -342901611.57979244) / Math.Sqrt((35022825392143.742 / (32)) + (13122716959031.328 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4716606547440858 = (138733288.86051968 - 342901611.57979244) / 138733288.86051968 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [   0,    0])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 342.1270554285714 > 152.00633104999997.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -166.01399594874204 (T) = (0 -344566859.94537044) / Math.Sqrt((34889472286940.195 / (32)) + (8145026019636.561 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4903309519532348 = (138361875.02513158 - 344566859.94537044) / 138361875.02513158 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEnd(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 343.8860604285714 > 152.62044103250003.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -252.5297018985524 (T) = (0 -344455439.9371388) / Math.Sqrt((16454589260956.13 / (32)) + (2495834036725.4434 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.4531846761248473 = (140411540.67587563 - 344455439.9371388) / 140411540.67587563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 370.79915553333336 > 188.38471239000003.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -139.98477749649882 (T) = (0 -379241048.1183049) / Math.Sqrt((27964385254483.887 / (32)) + (23309510553184.926 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.1910711059622583 = (173084774.4221211 - 379241048.1183049) / 173084774.4221211 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadToEndTests.ReadToEndAsync(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 381.33749753846155 > 187.773928825.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -182.8289364618816 (T) = (0 -383032128.79755807) / Math.Sqrt((16465271685911.637 / (32)) + (14160012543906.082 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.1952993922900443 = (174478310.40393767 - 383032128.79755807) / 174478310.40393767 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Tests.StreamReaderReadLineTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadLineAsync - Duration of single invocation 288.48 μs 364.97 μs 1.27 0.01 False
ReadLineAsync - Duration of single invocation 171.60 μs 249.46 μs 1.45 0.02 False
ReadLine - Duration of single invocation 120.75 μs 190.74 μs 1.58 0.03 False
ReadLine - Duration of single invocation 156.42 μs 224.42 μs 1.43 0.02 False
ReadLineAsync - Duration of single invocation 176.43 μs 253.62 μs 1.44 0.02 False
ReadLineAsync - Duration of single invocation 154.56 μs 232.45 μs 1.50 0.01 False
ReadLineAsync - Duration of single invocation 661.24 μs 734.13 μs 1.11 0.02 False
ReadLine - Duration of single invocation 297.42 μs 360.69 μs 1.21 0.03 False
ReadLine - Duration of single invocation 120.52 μs 196.48 μs 1.63 0.04 False
ReadLine - Duration of single invocation 123.40 μs 195.25 μs 1.58 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.StreamReaderReadLineTests*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 33, 128])

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 364.97431130160163 > 303.27525233364906.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -105.74528814905659 (T) = (0 -367661.41106679064) / Math.Sqrt((5655772.702884219 / (32)) + (7490484.239614789 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.28445670826981073 = (286238.8500131219 - 367661.41106679064) / 286238.8500131219 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 249.45796887133923 > 180.81255131610578.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -124.77162062622835 (T) = (0 -250127.60930457956) / Math.Sqrt((4490369.773671565 / (32)) + (4839082.117723512 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.46860482326317093 = (170316.48360571754 - 250127.60930457956) / 170316.48360571754 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 190.73753846993344 > 127.1518458442345.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -117.02457516230933 (T) = (0 -192147.97616663924) / Math.Sqrt((8358390.016987015 / (32)) + (2182611.7630159594 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.6042217171530597 = (119776.4461807908 - 192147.97616663924) / 119776.4461807908 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [  33,  128])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 224.42328868994332 > 164.28769215324516.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -136.32204675190678 (T) = (0 -224073.0196802743) / Math.Sqrt((4698215.324605978 / (32)) + (2069544.4353580296 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.45197656293437727 = (154322.75244679785 - 224073.0196802743) / 154322.75244679785 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [   0, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 253.61746688837482 > 184.70163169477905.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -97.67791735020535 (T) = (0 -253482.17224928158) / Math.Sqrt((3369275.92806918 / (32)) + (9837920.54078961 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.45161937977241834 = (174620.27290447304 - 253482.17224928158) / 174620.27290447304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 232.45217742438416 > 163.9383025758044.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -120.38491206991695 (T) = (0 -233720.3383482682) / Math.Sqrt((3215614.293333685 / (32)) + (5994402.971519742 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.513173117778234 = (154457.10447951636 - 233720.3383482682) / 154457.10447951636 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLineAsync(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 734.1251051051051 > 684.596769923127.
IsChangePoint: Marked as a change because one of 11/15/2022 5:29:43 PM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -38.89820728914121 (T) = (0 -738477.1814654428) / Math.Sqrt((77303101.64476955 / (32)) + (37548390.78064448 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12582070592771227 = (655945.6382150246 - 738477.1814654428) / 655945.6382150246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [   9,   32])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 360.68525176834663 > 310.1032737175708.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -46.68993184023283 (T) = (0 -360239.6195135139) / Math.Sqrt((36015044.18033246 / (32)) + (15750072.853910886 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.22446245765005543 = (294202.25770324795 - 360239.6195135139) / 294202.25770324795 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [1025, 2048])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 196.47523372734435 > 126.33300781890134.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -34.70089818897437 (T) = (0 -196296.95922701253) / Math.Sqrt((4922061.459330085 / (32)) + (86347355.26301076 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.6483713779393265 = (119085.39656421883 - 196296.95922701253) / 119085.39656421883 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.StreamReaderReadLineTests.ReadLine(LineLengthRange: [ 129, 1024])

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.24902956023936 > 128.28763265514368.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -175.76282677175627 (T) = (0 -194742.97860468423) / Math.Sqrt((4390441.610009334 / (32)) + (747278.4939185148 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.6169373632751914 = (120439.40787552965 - 194742.97860468423) / 120439.40787552965 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 35.50 μs 39.11 μs 1.10 0.01 False
DeserializeFromReader - Duration of single invocation 34.38 μs 36.97 μs 1.08 0.01 False
DeserializeFromStream - Duration of single invocation 36.36 μs 39.98 μs 1.10 0.01 False
DeserializeFromReader - Duration of single invocation 35.09 μs 38.08 μs 1.09 0.01 False
DeserializeFromString - Duration of single invocation 27.90 μs 30.62 μs 1.10 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 22.16 μs 25.39 μs 1.15 0.02 False
DeserializeFromString - Duration of single invocation 27.57 μs 30.17 μs 1.09 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 23.35 μs 25.97 μs 1.11 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromStream(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.11471088752665 > 37.571050221408235.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -44.91896339502226 (T) = (0 -39348.03820554488) / Math.Sqrt((104131.6458117231 / (32)) + (67554.8708264603 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10565696617390342 = (35587.9259203763 - 39348.03820554488) / 35587.9259203763 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 36.9688446060966 > 35.48951286052779.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 3:18:57 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -39.57981469020807 (T) = (0 -37139.80650446646) / Math.Sqrt((117377.29016407444 / (31)) + (60502.060954216235 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.09766565528369096 = (33835.26333878753 - 37139.80650446646) / 33835.26333878753 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.98175372965641 > 38.627694516786406.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.078243306372556 (T) = (0 -40057.29247487168) / Math.Sqrt((170460.3162253339 / (32)) + (50103.5462036025 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09902784457090134 = (36447.932300124245 - 40057.29247487168) / 36447.932300124245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 38.08170095818816 > 36.83765743002213.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.180816205760625 (T) = (0 -37664.10226500288) / Math.Sqrt((208289.93233267526 / (31)) + (70324.44485244896 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.09003865847440443 = (34552.996787945835 - 37664.10226500288) / 34552.996787945835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.6214580642531 > 29.379612776588118.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.169142813640562 (T) = (0 -30784.532281534248) / Math.Sqrt((167196.5229823251 / (32)) + (70163.35259058201 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10706044060089798 = (27807.454003888717 - 30784.532281534248) / 27807.454003888717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.389278424015004 > 23.80937102520421.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -49.07934990686807 (T) = (0 -25417.919577042758) / Math.Sqrt((76183.82574387964 / (32)) + (23044.810184790793 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13228655761222474 = (22448.30993193479 - 25417.919577042758) / 22448.30993193479 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.166039305286912 > 28.945141664366165.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.18371981796543 (T) = (0 -30336.528741542832) / Math.Sqrt((154356.96742146873 / (32)) + (102028.37378408326 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12191506287842242 = (27039.951370035473 - 30336.528741542832) / 27039.951370035473 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 25.96971780124735 > 24.69958277840184.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -34.39891015370049 (T) = (0 -26223.210696144626) / Math.Sqrt((113374.67792942697 / (32)) + (62457.59606129901 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12340784738734369 = (23342.556095838838 - 26223.210696144626) / 23342.556095838838 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Binary_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BinaryFormatter_ - Duration of single invocation 58.44 μs 63.78 μs 1.09 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Binary_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Binary_FromStream<LoginViewModel>.BinaryFormatter_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.77716136185243 > 61.48232250831575.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -18.533975188967663 (T) = (0 -63800.85349404777) / Math.Sqrt((987602.8579249766 / (32)) + (188904.81329797627 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06278618332064764 = (60031.692635205014 - 63800.85349404777) / 60031.692635205014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 8.33 μs 9.33 μs 1.12 0.02 False
DeserializeFromReader - Duration of single invocation 13.06 μs 14.43 μs 1.11 0.01 False
DeserializeFromStream - Duration of single invocation 19.38 μs 20.81 μs 1.07 0.01 False
DeserializeFromString - Duration of single invocation 11.43 μs 12.50 μs 1.09 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 7.94 μs 9.21 μs 1.16 0.01 False
DeserializeFromString - Duration of single invocation 11.65 μs 12.83 μs 1.10 0.01 False
DeserializeFromReader - Duration of single invocation 13.60 μs 14.43 μs 1.06 0.01 False
DeserializeFromStream - Duration of single invocation 19.78 μs 21.17 μs 1.07 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>.DeserializeFromUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.326230753610108 > 8.761068129208542.
IsChangePoint: Marked as a change because one of 12/8/2022 9:50:09 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.459605490108366 (T) = (0 -9408.663623622742) / Math.Sqrt((30097.86514128982 / (32)) + (4815.789554911094 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.15074259008027033 = (8176.167028776122 - 9408.663623622742) / 8176.167028776122 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.430565120036936 > 13.670373570894983.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.72691059852973 (T) = (0 -14460.85904147599) / Math.Sqrt((20181.042051716136 / (32)) + (29276.893517781253 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11636437171047351 = (12953.52969686709 - 14460.85904147599) / 12953.52969686709 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.81250068499622 > 20.29887981676191.
IsChangePoint: Marked as a change because one of 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.793192367755445 (T) = (0 -20835.492406158635) / Math.Sqrt((13286.005653631799 / (32)) + (21744.51946122354 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07884165753579138 = (19312.836374662704 - 20835.492406158635) / 19312.836374662704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.497089101473923 > 12.007813057001876.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.09976742025651 (T) = (0 -12518.683905900474) / Math.Sqrt((32701.86240240845 / (32)) + (10354.697688107248 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11417630671851209 = (11235.819529110864 - 12518.683905900474) / 11235.819529110864 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.209107311970861 > 8.41172422935002.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.66240413464441 (T) = (0 -9152.64913501208) / Math.Sqrt((25599.497995941023 / (32)) + (12467.890587078778 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.15992206747688853 = (7890.744897130295 - 9152.64913501208) / 7890.744897130295 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.826122859895403 > 12.275989018416922.
IsChangePoint: Marked as a change because one of 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.006845190505086 (T) = (0 -12748.011905398494) / Math.Sqrt((56237.219383702955 / (32)) + (10829.720264541296 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10998464742844576 = (11484.854258959725 - 12748.011905398494) / 11484.854258959725 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.432124614352782 > 14.196542709011444.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -19.360567505157128 (T) = (0 -14503.106642937624) / Math.Sqrt((104852.71192841549 / (32)) + (15494.652056218547 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09393218215952706 = (13257.774914626883 - 14503.106642937624) / 13257.774914626883 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.165136585912965 > 20.739903744831427.
IsChangePoint: Marked as a change because one of 12/8/2022 9:50:09 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.62157579098666 (T) = (0 -21112.25386765106) / Math.Sqrt((17218.46132240156 / (32)) + (16126.50287140397 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07236322663403622 = (19687.595903413057 - 21112.25386765106) / 19687.595903413057 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 701.56 μs 779.12 μs 1.11 0.01 False
DeserializeFromString - Duration of single invocation 631.55 μs 691.63 μs 1.10 0.01 False
DeserializeFromReader - Duration of single invocation 717.71 μs 793.45 μs 1.11 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 538.40 μs 613.72 μs 1.14 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 556.27 μs 628.77 μs 1.13 0.01 False
DeserializeFromStream - Duration of single invocation 596.91 μs 662.73 μs 1.11 0.01 False
DeserializeFromStream - Duration of single invocation 617.78 μs 691.13 μs 1.12 0.01 False
DeserializeFromString - Duration of single invocation 610.44 μs 676.00 μs 1.11 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromReader(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 779.1177470982142 > 740.0444833507526.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.387475332295413 (T) = (0 -775477.3294908953) / Math.Sqrt((172608102.91276294 / (32)) + (29622406.11626403 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11178708720888901 = (697505.2493528323 - 775477.3294908953) / 697505.2493528323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 691.6298252717389 > 656.807731496875.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -24.5548697319313 (T) = (0 -690079.0047758074) / Math.Sqrt((260007374.85173765 / (32)) + (21164164.409182217 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12172835273731306 = (615192.6204698603 - 690079.0047758074) / 615192.6204698603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 793.44684375 > 753.0726232315342.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.970252252339733 (T) = (0 -793067.7524486162) / Math.Sqrt((150697111.2571677 / (32)) + (45449447.764691584 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11736131314274736 = (709768.4008926293 - 793067.7524486162) / 709768.4008926293 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 613.720683392857 > 561.6590323545258.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -46.10047865868248 (T) = (0 -615757.2555271671) / Math.Sqrt((86311049.89733446 / (32)) + (10641345.789928606 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1570930242178104 = (532158.8175189425 - 615757.2555271671) / 532158.8175189425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 628.7675083928572 > 583.8697603401649.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -36.23958354688862 (T) = (0 -633831.9240810035) / Math.Sqrt((113064002.11106499 / (32)) + (31949616.24060184 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.15171516307773159 = (550337.3962596904 - 633831.9240810035) / 550337.3962596904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 662.7290402644231 > 625.6965644961168.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.750005474882755 (T) = (0 -673978.1921488835) / Math.Sqrt((94956280.1986989 / (32)) + (15572502.290183853 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.138005872683825 = (592244.9157133098 - 673978.1921488835) / 592244.9157133098 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 691.129070965719 > 646.3155744374999.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.430014055867446 (T) = (0 -693651.5344127642) / Math.Sqrt((118962728.75412099 / (32)) + (37997858.91492529 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.14065883915520005 = (608114.811021409 - 693651.5344127642) / 608114.811021409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 676.0003925585283 > 643.3692677812498.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.43194498337497 (T) = (0 -672097.1020948286) / Math.Sqrt((283242594.6785972 / (32)) + (23907224.10440074 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11917843635330382 = (600527.2084090263 - 672097.1020948286) / 600527.2084090263 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 245.58 μs 300.10 μs 1.22 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 207.85 μs 265.29 μs 1.28 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 215.18 μs 263.70 μs 1.23 0.05 False
DeserializeFromString - Duration of single invocation 249.84 μs 298.23 μs 1.19 0.04 False
DeserializeFromStream - Duration of single invocation 241.09 μs 303.43 μs 1.26 0.02 False
DeserializeFromReader - Duration of single invocation 292.66 μs 355.95 μs 1.22 0.01 False
DeserializeFromReader - Duration of single invocation 300.51 μs 344.93 μs 1.15 0.01 False
DeserializeFromString - Duration of single invocation 247.95 μs 304.24 μs 1.23 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 300.101385731456 > 255.00648860817313.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -45.25115998528723 (T) = (0 -304404.5850580337) / Math.Sqrt((35597297.487133875 / (32)) + (11607723.716027515 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.24542423455272044 = (244418.38902175936 - 304404.5850580337) / 244418.38902175936 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 265.28620818452384 > 220.16697160578545.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -48.823275739874546 (T) = (0 -266719.6377888186) / Math.Sqrt((27588144.43911232 / (32)) + (4783743.878222595 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.24131746216291933 = (214868.19119105604 - 266719.6377888186) / 214868.19119105604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 263.69785543785315 > 224.32491127782538.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.166315270327125 (T) = (0 -265366.1315753623) / Math.Sqrt((30103767.277808044 / (31)) + (12630582.990281353 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.221216284596741 = (217296.58777272946 - 265366.1315753623) / 217296.58777272946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 298.22573199685536 > 264.3197422572245.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.71085073751277 (T) = (0 -301959.9436534542) / Math.Sqrt((54862937.12147231 / (32)) + (9673936.283558825 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.19411759029968156 = (252872.87123680412 - 301959.9436534542) / 252872.87123680412 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 303.4283639627659 > 253.93822408691406.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -49.486751928717176 (T) = (0 -304322.6128591171) / Math.Sqrt((24355352.77368365 / (32)) + (12336841.994829487 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.2431228130353664 = (244804.94579296184 - 304322.6128591171) / 244804.94579296184 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 355.94936685606064 > 311.8077715064859.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -36.513115922410286 (T) = (0 -348050.95153814706) / Math.Sqrt((39368404.313817166 / (32)) + (10535809.996390192 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.16462708965068415 = (298851.8424747794 - 348050.95153814706) / 298851.8424747794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 344.92869206349206 > 314.7460280660378.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.61723364380348 (T) = (0 -348155.66070935514) / Math.Sqrt((40741532.41023196 / (32)) + (21127401.295375295 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1655774542472476 = (298698.00538841134 - 348155.66070935514) / 298698.00538841134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 304.24205735544217 > 264.03092151575856.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.936863675190715 (T) = (0 -304014.17097560235) / Math.Sqrt((36777175.56674664 / (32)) + (15640394.816101173 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.20182190329835314 = (252961.083619168 - 304014.17097560235) / 252961.083619168 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Tests.Perf_Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 1.25 μs 3.38 μs 2.70 0.16 False
GetString - Duration of single invocation 219.69 ns 335.74 ns 1.53 0.11 False
GetChars - Duration of single invocation 1.26 μs 3.42 μs 2.71 0.19 False
GetChars - Duration of single invocation 568.53 ns 2.69 μs 4.73 0.24 False
GetString - Duration of single invocation 213.72 ns 319.97 ns 1.50 0.18 False
GetChars - Duration of single invocation 327.34 ns 476.61 ns 1.46 0.15 False
GetChars - Duration of single invocation 251.56 ns 344.32 ns 1.37 0.14 False
GetString - Duration of single invocation 548.50 ns 2.62 μs 4.78 0.26 False
GetByteCount - Duration of single invocation 176.07 ns 205.02 ns 1.16 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "utf-8")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.3789843037652028 > 1.3093145495720204.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -314.8498003809899 (T) = (0 -3389.0248389764956) / Math.Sqrt((1201.7856215382842 / (32)) + (188.45238651097472 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.8075645466022743 = (1207.1048706886925 - 3389.0248389764956) / 1207.1048706886925 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "ascii")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 335.73963564974764 > 232.40825653608348.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.863074831927094 (T) = (0 -334.79790017174605) / Math.Sqrt((17.969470215850315 / (32)) + (146.64104808210638 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.5008997300660629 = (223.0648013755121 - 334.79790017174605) / 223.0648013755121 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "utf-8")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.417482040607654 > 1.3244500624788713.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -246.45758075913972 (T) = (0 -3396.3214305839956) / Math.Sqrt((1853.1427395522605 / (32)) + (363.9819993470548 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.7888374828191989 = (1217.8269445628289 - 3396.3214305839956) / 1217.8269445628289 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 512, encName: "ascii")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.690125716838193 > 595.5916739478217.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -275.0194760779596 (T) = (0 -2673.589788969517) / Math.Sqrt((1839.6845809039244 / (32)) + (60.000425982047226 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -4.056637537525866 = (528.7287785862665 - 2673.589788969517) / 528.7287785862665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 16, encName: "utf-8")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 319.96951344785367 > 218.78762380497116.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -49.71297476073591 (T) = (0 -319.947320916381) / Math.Sqrt((49.87088092012066 / (32)) + (66.68556783580337 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.5539172173895163 = (205.8972751803809 - 319.947320916381) / 205.8972751803809 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "utf-8")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 476.61089806233315 > 340.46227347337253.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.25169565219231 (T) = (0 -466.83465553922986) / Math.Sqrt((86.97275671645143 / (32)) + (188.88783580214394 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.45645348188357665 = (320.5283665740495 - 466.83465553922986) / 320.5283665740495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetChars(size: 16, encName: "ascii")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 344.3194367894217 > 249.78788424575143.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -43.52796084273208 (T) = (0 -349.8949363119095) / Math.Sqrt((30.401788447443217 / (32)) + (101.51058240543992 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.4691696673748651 = (238.15829041522966 - 349.8949363119095) / 238.15829041522966 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetString(size: 512, encName: "ascii")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.624138092430368 > 579.5206245130424.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -275.7549487000805 (T) = (0 -2624.92501639789) / Math.Sqrt((1764.221839209932 / (32)) + (63.92374913814856 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -4.1217611109963395 = (512.5043826745879 - 2624.92501639789) / 512.5043826745879 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Tests.Perf_Encoding.GetByteCount(size: 16, encName: "utf-8")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 205.0181337048092 > 185.81782259994327.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/2/2022 10:04:21 AM, 12/21/2022 3:18:57 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -14.116905115604025 (T) = (0 -198.77390995518272) / Math.Sqrt((23.885265124888445 / (31)) + (18.308416821572152 / (19))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (19) - 2, .025) and -0.10317140437319937 = (180.1840667435739 - 198.77390995518272) / 180.1840667435739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 614.97 μs 668.87 μs 1.09 0.02 False
DeserializeFromString - Duration of single invocation 610.43 μs 677.18 μs 1.11 0.01 False
DeserializeFromReader - Duration of single invocation 660.51 μs 722.01 μs 1.09 0.02 False
DeserializeFromReader - Duration of single invocation 665.17 μs 721.69 μs 1.08 0.01 False
DeserializeFromStream - Duration of single invocation 621.60 μs 679.52 μs 1.09 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 589.92 μs 639.79 μs 1.08 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 595.00 μs 640.85 μs 1.08 0.02 False
DeserializeFromStream - Duration of single invocation 613.69 μs 681.57 μs 1.11 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>.DeserializeFromString(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 668.8749429347826 > 647.9810253875.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -10.770847846824743 (T) = (0 -670908.7533219197) / Math.Sqrt((464803988.92042226 / (32)) + (40585467.007921755 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07039205466617518 = (626787.8674895032 - 670908.7533219197) / 626787.8674895032 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 677.1796290760868 > 645.9602352439905.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -11.246581680970248 (T) = (0 -670459.6546966173) / Math.Sqrt((439913431.8195788 / (32)) + (30446946.902906116 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07056121552207681 = (626269.3295587554 - 670459.6546966173) / 626269.3295587554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 722.0125936355312 > 699.7732565452551.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -11.76545071502406 (T) = (0 -717127.2754986252) / Math.Sqrt((367033199.50225747 / (32)) + (54153602.27957543 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06658344575501828 = (672359.277985026 - 717127.2754986252) / 672359.277985026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 721.6926679421769 > 707.6665971206976.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -12.152139772152902 (T) = (0 -718228.0738843594) / Math.Sqrt((346819626.9401859 / (32)) + (41803893.85429487 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06539396374012522 = (674143.1792639218 - 718228.0738843594) / 674143.1792639218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 679.5160030797101 > 639.3910576947115.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -15.746456456035162 (T) = (0 -675477.3883190022) / Math.Sqrt((313916744.40939265 / (32)) + (42055546.319821626 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08842893936767077 = (620598.5194691946 - 675477.3883190022) / 620598.5194691946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 639.7859040464743 > 618.9263271935096.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -14.341228287792477 (T) = (0 -642009.0377794956) / Math.Sqrt((275130110.6522005 / (32)) + (28924874.837100297 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07684196028262803 = (596196.1564081268 - 642009.0377794956) / 596196.1564081268 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 640.8519809027779 > 615.4936242739667.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -12.966618861887012 (T) = (0 -642240.0538052747) / Math.Sqrt((300250326.18648976 / (32)) + (70917112.03110993 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07955542723407831 = (594911.6067627519 - 642240.0538052747) / 594911.6067627519 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 681.5741112012988 > 639.9625060312501.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -14.82776851757898 (T) = (0 -672662.1536796738) / Math.Sqrt((320366091.1039172 / (32)) + (47344512.343678996 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08504082588284018 = (619941.7917131038 - 672662.1536796738) / 619941.7917131038 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 4.64 μs 5.43 μs 1.17 0.01 False
SerializeToString - Duration of single invocation 4.83 μs 5.39 μs 1.12 0.02 False
SerializeObjectProperty - Duration of single invocation 10.94 μs 11.74 μs 1.07 0.01 False
SerializeObjectProperty - Duration of single invocation 10.76 μs 11.58 μs 1.08 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Nullable<DateTimeOffset>>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.434032139981595 > 4.922039112977625.
IsChangePoint: Marked as a change because one of 12/6/2022 11:13:59 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.75073517925273 (T) = (0 -5484.534910594264) / Math.Sqrt((3879.2370483988757 / (32)) + (4830.055380288761 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1768159687660179 = (4660.486478905636 - 5484.534910594264) / 4660.486478905636 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.392731282271945 > 4.863728971847037.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -46.89809631279048 (T) = (0 -5489.213733259637) / Math.Sqrt((4728.734797278206 / (32)) + (2953.9233129099116 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1776932311468364 = (4660.987758173872 - 5489.213733259637) / 4660.987758173872 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.741324822664106 > 11.473151591996626.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -19.07030850905867 (T) = (0 -11851.785739146935) / Math.Sqrt((14975.186566882203 / (32)) + (29933.54531078299 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08023796875333945 = (10971.458217511665 - 11851.785739146935) / 10971.458217511665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.577462066798942 > 11.244608011289895.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.061368736619897 (T) = (0 -11562.790710300593) / Math.Sqrt((12401.804466274509 / (32)) + (9336.551175487282 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07881679592790497 = (10718.029932371677 - 11562.790710300593) / 10718.029932371677 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 614.76 μs 673.56 μs 1.10 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 631.74 μs 684.72 μs 1.08 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 673.5580556159418 > 644.2387186897189.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.950813838609722 (T) = (0 -673778.5541256464) / Math.Sqrt((337444837.67204237 / (32)) + (7343842.886732819 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12705619496410786 = (597821.6145177246 - 673778.5541256464) / 597821.6145177246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 684.7215368788818 > 661.3318971562502.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -21.952902393068793 (T) = (0 -691182.9462426325) / Math.Sqrt((335210926.414993 / (32)) + (26091596.902240362 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12319115202050876 = (615374.2797913457 - 691182.9462426325) / 615374.2797913457 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Load - Duration of single invocation 851.43 μs 949.27 μs 1.11 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Configuration.Xml.XmlConfigurationProviderBenchmarks.Load(FileName: "deep.xml")

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 949.2749779776675 > 896.6148272390737.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.44729935883487 (T) = (0 -946003.1702464581) / Math.Sqrt((215622642.16733244 / (32)) + (24218931.547177643 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10805709772750122 = (853749.4793243081 - 946003.1702464581) / 853749.4793243081 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
JsonNet_ - Duration of single invocation 1.08 ms 1.15 ms 1.06 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 608.76 μs 687.73 μs 1.13 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 588.90 μs 684.03 μs 1.16 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.JsonNet_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.153918313034188 > 1.136902916700721.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 11/16/2022 11:42:33 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -21.581594316338364 (T) = (0 -1151030.5268942346) / Math.Sqrt((163318650.80637664 / (32)) + (53606605.41456375 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.05630424921046776 = (1089677.077181664 - 1151030.5268942346) / 1089677.077181664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 687.7256945652174 > 640.2437561829697.
IsChangePoint: Marked as a change because one of 12/17/2022 9:10:43 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.631861003368954 (T) = (0 -683537.382562385) / Math.Sqrt((154485739.00235075 / (32)) + (24992474.31624123 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13510674230676734 = (602178.94677755 - 683537.382562385) / 602178.94677755 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 684.0326290760869 > 615.7146998900463.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -38.39344851224965 (T) = (0 -674472.6226352006) / Math.Sqrt((83687509.80294928 / (32)) + (51980206.39478273 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.15411404647486784 = (584407.2556740068 - 674472.6226352006) / 584407.2556740068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 1.79 μs 2.13 μs 1.19 0.09 False
SerializeObjectProperty - Duration of single invocation 8.18 μs 8.81 μs 1.08 0.01 False
SerializeToString - Duration of single invocation 1.81 μs 1.98 μs 1.10 0.05 False
SerializeObjectProperty - Duration of single invocation 7.93 μs 8.66 μs 1.09 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.1270622757872606 > 1.8729069573493782.
IsChangePoint: Marked as a change because one of 12/7/2022 5:17:51 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -13.22473849198822 (T) = (0 -2000.6101113513012) / Math.Sqrt((578.236925939743 / (32)) + (4040.85765993872 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11476790944101063 = (1794.6427183703981 - 2000.6101113513012) / 1794.6427183703981 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.81114929966044 > 8.56904023747293.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.865459800597012 (T) = (0 -8899.154916490363) / Math.Sqrt((7687.995911002725 / (32)) + (14483.546421192264 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09057816884985767 = (8160.033980760465 - 8899.154916490363) / 8160.033980760465 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.983533450979151 > 1.872675415491977.
IsChangePoint: Marked as a change because one of 12/6/2022 11:39:16 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -12.266050825415062 (T) = (0 -1979.0466411386633) / Math.Sqrt((625.9598115034448 / (32)) + (3207.616037509923 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09548297305774152 = (1806.5517126338304 - 1979.0466411386633) / 1806.5517126338304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.662408700166115 > 8.317559973869617.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -23.50848688304306 (T) = (0 -8590.888056321062) / Math.Sqrt((9661.562515644828 / (32)) + (10874.499447055938 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08976309895476306 = (7883.262026912949 - 8590.888056321062) / 7883.262026912949 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 11.31 μs 12.49 μs 1.10 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 11.43 μs 12.60 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.491231593078759 > 12.072258497215214.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.02819791685039 (T) = (0 -12530.836558939056) / Math.Sqrt((35501.54076754799 / (32)) + (6099.932001123433 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10775203979025842 = (11311.950787571235 - 12530.836558939056) / 11311.950787571235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.59648244264332 > 11.982601174943467.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.554966706639355 (T) = (0 -12506.644215506176) / Math.Sqrt((48830.01661197086 / (32)) + (9349.268491184615 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11514797724105728 = (11215.23283972443 - 12506.644215506176) / 11215.23283972443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 26.06 μs 34.52 μs 1.32 0.03 False
SerializeToString - Duration of single invocation 18.10 μs 24.69 μs 1.36 0.05 False
SerializeToString - Duration of single invocation 20.64 μs 27.26 μs 1.32 0.03 False
SerializeObjectProperty - Duration of single invocation 26.21 μs 33.20 μs 1.27 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<BinaryData>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.51745451713984 > 27.36500307823989.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -50.36946230231341 (T) = (0 -33187.0729016171) / Math.Sqrt((60312.13298411117 / (32)) + (315145.2866813938 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.268002637958 = (26172.715977201584 - 33187.0729016171) / 26172.715977201584 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 24.694523382936502 > 18.9928747201444.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -51.75533706805692 (T) = (0 -24914.221319300974) / Math.Sqrt((49959.97053629956 / (32)) + (272489.45036493405 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.3669568944227126 = (18226.04752275136 - 24914.221319300974) / 18226.04752275136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.261826671803803 > 21.636161021368526.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -47.119661864240186 (T) = (0 -27390.648865150048) / Math.Sqrt((189993.74208042765 / (32)) + (237957.61354361172 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.31251400877269886 = (20868.843061539894 - 27390.648865150048) / 20868.843061539894 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;BinaryData&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.196139384920635 > 27.516349692243654.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -51.751920629779896 (T) = (0 -33429.080962650136) / Math.Sqrt((69292.07031408974 / (32)) + (296188.2698982639 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.2678276184318316 = (26367.21307901335 - 33429.080962650136) / 26367.21307901335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Perf_Utf8Encoding

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetString - Duration of single invocation 397.22 μs 1.02 ms 2.58 0.17 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Utf8Encoding*'

Payloads

Baseline
Compare

Histogram

System.Text.Perf_Utf8Encoding.GetString(Input: EnglishAllAscii)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0235455231584822 > 417.3023379071514.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -206.7278979410833 (T) = (0 -1021356.550871169) / Math.Sqrt((242854737.0910022 / (32)) + (38527045.368192725 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -1.712589988036143 = (376524.48596207093 - 1021356.550871169) / 376524.48596207093 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CaseInsensitiveMatching - Duration of single invocation 27.47 μs 30.14 μs 1.10 0.00 False
CaseInsensitiveNotMatching - Duration of single invocation 27.17 μs 30.26 μs 1.11 0.01 False
Baseline - Duration of single invocation 27.39 μs 30.38 μs 1.11 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive<Location>.CaseInsensitiveMatching

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.139358223732685 > 28.759150578217017.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/8/2022 6:11:08 PM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -45.456252849698586 (T) = (0 -30322.412923999964) / Math.Sqrt((72240.53739302911 / (32)) + (49174.12860184143 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1184305101481531 = (27111.57523768133 - 30322.412923999964) / 27111.57523768133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;.CaseInsensitiveNotMatching

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.260562832900845 > 28.749435030058606.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/8/2022 11:25:18 AM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.226482363004685 (T) = (0 -30283.800542892048) / Math.Sqrt((83991.89268372409 / (32)) + (103440.71972739763 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11910505296804681 = (27060.73077105186 - 30283.800542892048) / 27060.73077105186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;.Baseline

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.382765281791908 > 28.76182038989015.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/11/2022 6:23:40 AM, 11/15/2022 8:48:03 PM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.32700307425147 (T) = (0 -30166.70453125157) / Math.Sqrt((109360.64635875795 / (32)) + (34885.17863390658 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11142700959647461 = (27142.31728289938 - 30166.70453125157) / 27142.31728289938 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 29.24 μs 32.39 μs 1.11 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 30.17 μs 33.20 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<Location>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 32.38915192207112 > 30.930936191148476.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.937838741151175 (T) = (0 -32507.460378145373) / Math.Sqrt((114766.41746538058 / (32)) + (70644.579276651 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.112526009379817 = (29219.505974756325 - 32507.460378145373) / 29219.505974756325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;Location&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 33.19773621963563 > 31.566908183956382.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -45.13859130784037 (T) = (0 -33334.334642909394) / Math.Sqrt((79820.56902051633 / (32)) + (60412.018302353645 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11554290630760781 = (29881.71450369794 - 33334.334642909394) / 29881.71450369794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Binary_FromStream<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BinaryFormatter_ - Duration of single invocation 100.41 μs 110.38 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Binary_FromStream&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Binary_FromStream<Location>.BinaryFormatter_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 110.38420301573424 > 105.6316854444152.
IsChangePoint: Marked as a change because one of 12/9/2022 1:52:32 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -24.500457016401594 (T) = (0 -109646.37550829435) / Math.Sqrt((973383.7629521537 / (32)) + (1156703.8116356665 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.073831952923858 = (102107.57391763796 - 109646.37550829435) / 102107.57391763796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 15.58 μs 16.71 μs 1.07 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<LargeStructWithProperties>.DeserializeFromUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.706873672400896 > 16.372884292513252.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -17.397343909427484 (T) = (0 -16673.02721423776) / Math.Sqrt((57742.57763965338 / (32)) + (50753.66594654982 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07637398714617304 = (15489.99456819235 - 16673.02721423776) / 15489.99456819235 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
CachedJsonSerializerContext - Duration of single invocation 19.96 μs 21.48 μs 1.08 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ColdStartSerialization&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ColdStartSerialization<SimpleStructWithProperties>.CachedJsonSerializerContext

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.480601628877693 > 21.120830948869543.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -33.61888616863982 (T) = (0 -21310.87360555027) / Math.Sqrt((26686.93822019572 / (32)) + (14412.486635303942 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06812691331838366 = (19951.630597288393 - 21310.87360555027) / 19951.630597288393 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 27.24 μs 30.07 μs 1.10 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 27.55 μs 30.36 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<Location>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.07220815442112 > 28.734605883211433.
IsChangePoint: Marked as a change because one of 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.8439258465099 (T) = (0 -30289.661399596098) / Math.Sqrt((125641.34801454449 / (32)) + (26794.86778671536 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11015711767834857 = (27284.121244874164 - 30289.661399596098) / 27284.121244874164 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.355440228435945 > 29.23408225317602.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.375263762886597 (T) = (0 -30435.634910668778) / Math.Sqrt((189579.08663771048 / (32)) + (93227.25902251678 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10470167204762326 = (27550.999225206848 - 30435.634910668778) / 27550.999225206848 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Net.Tests.Perf_WebUtility

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Decode_DecodingRequired - Duration of single invocation 1.32 μs 1.49 μs 1.13 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Net.Tests.Perf_WebUtility*'

Payloads

Baseline
Compare

Histogram

System.Net.Tests.Perf_WebUtility.Decode_DecodingRequired

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.494903421529078 > 1.3852946125767134.
IsChangePoint: Marked as a change because one of 11/15/2022 8:48:03 PM, 12/7/2022 12:36:41 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.152627758366716 (T) = (0 -1502.2221631022708) / Math.Sqrt((371.6301234693858 / (32)) + (531.9006915608163 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1422157616234688 = (1315.1824844082987 - 1502.2221631022708) / 1315.1824844082987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Hashtable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 569.71 μs 637.44 μs 1.12 0.01 False
DeserializeFromReader - Duration of single invocation 656.45 μs 707.42 μs 1.08 0.01 False
DeserializeFromString - Duration of single invocation 627.48 μs 661.48 μs 1.05 0.01 False
DeserializeFromStream - Duration of single invocation 614.71 μs 680.44 μs 1.11 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 563.49 μs 633.45 μs 1.12 0.01 False
DeserializeFromStream - Duration of single invocation 615.37 μs 668.96 μs 1.09 0.01 False
DeserializeFromString - Duration of single invocation 620.07 μs 670.97 μs 1.08 0.01 False
DeserializeFromReader - Duration of single invocation 646.48 μs 726.54 μs 1.12 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Hashtable>.DeserializeFromUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 637.4444927455359 > 603.6425031032987.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.876868370969074 (T) = (0 -636576.7165650106) / Math.Sqrt((88196823.95556255 / (32)) + (58061346.19207603 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1199521263941827 = (568396.3640611533 - 636576.7165650106) / 568396.3640611533 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 707.4224500811689 > 689.7347252451315.
IsChangePoint: Marked as a change because one of 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.464672330500285 (T) = (0 -717934.0257673792) / Math.Sqrt((122790862.72138733 / (32)) + (40863304.318772115 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10022894603504624 = (652531.4829741904 - 717934.0257673792) / 652531.4829741904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 661.4795749223603 > 647.7067918125001.
IsChangePoint: Marked as a change because one of 12/8/2022 3:49:24 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -23.168941778515318 (T) = (0 -668433.4820535815) / Math.Sqrt((183733986.268912 / (32)) + (36266171.04447282 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10684848203576494 = (603906.9420090534 - 668433.4820535815) / 603906.9420090534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 680.4444509963768 > 640.4054564350962.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.158732831563686 (T) = (0 -678766.9550813745) / Math.Sqrt((99725272.62396666 / (32)) + (36513031.35406228 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11622513515934793 = (608091.444728555 - 678766.9550813745) / 608091.444728555 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 633.4472068452382 > 594.2031366319444.
IsChangePoint: Marked as a change because one of 12/8/2022 5:23:10 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.741614591236754 (T) = (0 -636370.4922061648) / Math.Sqrt((75245441.1142918 / (32)) + (49386970.19005954 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13139621882699204 = (562464.7507359892 - 636370.4922061648) / 562464.7507359892 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 668.95683340301 > 650.9295480000001.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.427342714004702 (T) = (0 -676340.5288536268) / Math.Sqrt((120252640.04625684 / (32)) + (45350810.42557033 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1085231013450018 = (610127.5905148067 - 676340.5288536268) / 610127.5905148067 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 670.9659904891303 > 642.7646173197115.
IsChangePoint: Marked as a change because one of 12/7/2022 12:36:41 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -38.380926461122634 (T) = (0 -674957.1051319931) / Math.Sqrt((62149380.55711989 / (32)) + (23935225.81592887 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1146511610426407 = (605532.1420027416 - 674957.1051319931) / 605532.1420027416 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 726.5392595663266 > 684.6463583483572.
IsChangePoint: Marked as a change because one of 12/7/2022 8:33:47 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -32.6366074640715 (T) = (0 -721325.6932979912) / Math.Sqrt((51097960.52748772 / (32)) + (67041843.018863074 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11653524268331984 = (646039.3418164401 - 721325.6932979912) / 646039.3418164401 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Tests.Perf_File

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadAllLines - Duration of single invocation 102.17 μs 125.84 μs 1.23 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_File*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_File.ReadAllLines

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 125.84303388637507 > 106.03872424721303.
IsChangePoint: Marked as a change because one of 11/7/2022 4:10:19 PM, 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -19.98857662447957 (T) = (0 -124520.549423776) / Math.Sqrt((31863839.93024228 / (32)) + (802770.8893861345 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.19578655312077878 = (104132.75604981567 - 124520.549423776) / 104132.75604981567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 7.42 μs 8.25 μs 1.11 0.01 False
SerializeObjectProperty - Duration of single invocation 13.44 μs 14.37 μs 1.07 0.01 False
SerializeToString - Duration of single invocation 4.77 μs 5.50 μs 1.15 0.01 False
SerializeObjectProperty - Duration of single invocation 12.33 μs 13.46 μs 1.09 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.246067982340538 > 7.879399919556919.
IsChangePoint: Marked as a change because one of 12/7/2022 6:04:36 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.822227819518748 (T) = (0 -8252.127526832795) / Math.Sqrt((6744.363986145201 / (32)) + (7739.389088528335 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10441610437935914 = (7471.936975665693 - 8252.127526832795) / 7471.936975665693 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.372311462842713 > 13.88439107625636.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -21.93679775996491 (T) = (0 -14437.058298332799) / Math.Sqrt((26377.41545028961 / (32)) + (38742.93366359005 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09039499934514979 = (13240.209563509694 - 14437.058298332799) / 13240.209563509694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 5.49846535668078 > 5.049445997419509.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.08122011983471 (T) = (0 -5563.559307839152) / Math.Sqrt((3767.4889752198565 / (32)) + (3579.8937196829706 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.15126282937802651 = (4832.570952407873 - 5563.559307839152) / 4832.570952407873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;.SerializeObjectProperty(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.46393074799197 > 12.9631675060635.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.880220625832465 (T) = (0 -13465.903989506927) / Math.Sqrt((13059.373467795613 / (32)) + (34157.66868438256 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08883703022913832 = (12367.235514274455 - 13465.903989506927) / 12367.235514274455 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_ToString<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 6.68 μs 7.67 μs 1.15 0.02 False
SystemTextJson_Reflection_ - Duration of single invocation 10.60 μs 11.66 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<LoginViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.670073255670611 > 7.113944295349784.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.67521095743716 (T) = (0 -7733.271580163701) / Math.Sqrt((21148.751180960015 / (32)) + (13555.061652891174 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.16862587752683142 = (6617.405731704026 - 7733.271580163701) / 6617.405731704026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_ToString&lt;LoginViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.663419516260781 > 11.113926270384193.
IsChangePoint: Marked as a change because one of 12/7/2022 8:33:47 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -46.16277212219076 (T) = (0 -11749.614811308402) / Math.Sqrt((13086.847049056198 / (32)) + (6930.554481455579 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12448942119869419 = (10448.844239711418 - 11749.614811308402) / 10448.844239711418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 6.92 μs 7.37 μs 1.06 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<SimpleStructWithProperties>.DeserializeFromUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.370749449872894 > 7.2520005150868885.
IsChangePoint: Marked as a change because one of 12/8/2022 9:08:34 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -15.314130705185233 (T) = (0 -7345.814087847136) / Math.Sqrt((13079.856768897867 / (32)) + (9937.973344524995 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06908667565417208 = (6871.111814532948 - 7345.814087847136) / 6871.111814532948 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 144.80 μs 180.05 μs 1.24 0.04 False
DeserializeFromReader - Duration of single invocation 169.82 μs 198.87 μs 1.17 0.03 False
DeserializeFromReader - Duration of single invocation 166.96 μs 196.95 μs 1.18 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 126.82 μs 153.96 μs 1.21 0.02 False
DeserializeFromString - Duration of single invocation 143.75 μs 170.18 μs 1.18 0.04 False
DeserializeFromString - Duration of single invocation 146.53 μs 171.39 μs 1.17 0.04 False
DeserializeFromUtf8Bytes - Duration of single invocation 121.12 μs 151.65 μs 1.25 0.02 False
DeserializeFromStream - Duration of single invocation 146.07 μs 178.78 μs 1.22 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromStream(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 180.0483097940613 > 151.63295808946108.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.6052024513921 (T) = (0 -175640.61055416128) / Math.Sqrt((20644907.907701965 / (32)) + (4663700.336206065 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.18324818086116448 = (148439.3666477734 - 175640.61055416128) / 148439.3666477734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 198.86832491987178 > 178.71009426204932.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.675107360077405 (T) = (0 -196016.16872048835) / Math.Sqrt((19483444.500577707 / (32)) + (2735338.697829183 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.14044747051445391 = (171876.5430134768 - 196016.16872048835) / 171876.5430134768 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 196.9505663426763 > 176.98162653057796.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.852860196019098 (T) = (0 -195195.55964391446) / Math.Sqrt((20216781.821573723 / (32)) + (3411834.9290550253 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.14243193209528932 = (170859.68464301762 - 195195.55964391446) / 170859.68464301762 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 153.96015918367348 > 130.85169329175068.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.167269045440282 (T) = (0 -152123.83973825976) / Math.Sqrt((17920770.566145346 / (32)) + (3077178.536034163 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.19606796012586536 = (127186.61882912688 - 152123.83973825976) / 127186.61882912688 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 170.18012422360246 > 151.49654927034126.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -23.82454406389597 (T) = (0 -171861.86659433323) / Math.Sqrt((26556460.55366215 / (32)) + (3666234.802506612 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.16405449982662706 = (147640.73900313958 - 171861.86659433323) / 147640.73900313958 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.38828874547102 > 153.4771208366737.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -23.285171060626933 (T) = (0 -171244.22423835314) / Math.Sqrt((25951911.03723177 / (32)) + (5015801.47319883 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.16542090186021055 = (146937.66343560352 - 171244.22423835314) / 146937.66343560352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 151.65248994391024 > 130.00292647988627.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.288554979887966 (T) = (0 -152276.34026687566) / Math.Sqrt((21158385.875148606 / (32)) + (3082845.1168486555 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.19546496508645705 = (127378.33789705657 - 152276.34026687566) / 127378.33789705657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 178.78284597378277 > 152.56847647146037.
IsChangePoint: Marked as a change because one of 12/16/2022 11:20:37 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.671248009334438 (T) = (0 -176766.74059163066) / Math.Sqrt((17091347.96970714 / (32)) + (4426258.2766021015 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.18798596336913684 = (148795.3107546144 - 176766.74059163066) / 148795.3107546144 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 13.54 μs 14.67 μs 1.08 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 13.66 μs 14.78 μs 1.08 0.01 False
JsonNet_ - Duration of single invocation 25.48 μs 27.14 μs 1.07 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.67416542542509 > 14.220186211739394.
IsChangePoint: Marked as a change because one of 12/7/2022 8:33:47 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -43.148521716288414 (T) = (0 -14836.322710688193) / Math.Sqrt((8909.95530082964 / (32)) + (12701.952279840069 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10039393578634659 = (13482.73761622113 - 14836.322710688193) / 13482.73761622113 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.78244865972937 > 14.265037374669344.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -37.458966429239894 (T) = (0 -14776.315262204062) / Math.Sqrt((19030.230934534935 / (32)) + (10730.141903383697 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09586706459515719 = (13483.674926997492 - 14776.315262204062) / 13483.674926997492 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;.JsonNet_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.143591724738677 > 26.847655590361583.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -21.40753277906109 (T) = (0 -27175.263050143403) / Math.Sqrt((47435.27501902192 / (32)) + (57010.634931876455 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.05676484747707574 = (25715.525185211947 - 27175.263050143403) / 25715.525185211947 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Tests.Perf_Directory

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
EnumerateFiles - Duration of single invocation 16.65 ms 17.76 ms 1.07 0.04 False
GetCurrentDirectory - Duration of single invocation 2.09 μs 2.80 μs 1.34 0.15 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_Directory*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Directory.EnumerateFiles

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.76238604081633 > 17.465562767333335.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/15/2022 10:02:59 PM, 12/2/2022 5:47:06 PM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -18.696317844441612 (T) = (0 -17707088.309167683) / Math.Sqrt((84969259301.19229 / (32)) + (36310472298.07583 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07778917375759324 = (16429083.479688212 - 17707088.309167683) / 16429083.479688212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.IO.Tests.Perf_Directory.GetCurrentDirectory

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.8028510769978623 > 2.2144309304532577.
IsChangePoint: Marked as a change because one of 11/16/2022 11:42:33 AM, 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -52.12794443993509 (T) = (0 -2810.054299807295) / Math.Sqrt((5649.2114691172 / (32)) + (590.4657124963286 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.3668683546392663 = (2055.833899636153 - 2810.054299807295) / 2055.833899636153 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ArrayList>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 161.31 μs 171.30 μs 1.06 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ArrayList&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ArrayList>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 171.3004583786232 > 169.77053780723128.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -11.517120052945453 (T) = (0 -171327.2229186811) / Math.Sqrt((31183021.192064915 / (32)) + (3927071.3093500435 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.07922932363198235 = (158749.59952172663 - 171327.2229186811) / 158749.59952172663 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 3.81 ms 4.06 ms 1.07 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 3.82 ms 4.07 ms 1.06 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.062599317777778 > 4.021197670052083.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -23.058852147684473 (T) = (0 -4099184.889418682) / Math.Sqrt((835667807.3299638 / (32)) + (1715692106.1303177 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.0660839663732525 = (3845086.333456303 - 4099184.889418682) / 3845086.333456303 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.070506162222222 > 3.9796619841730765.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -17.50846006696021 (T) = (0 -4082895.336597716) / Math.Sqrt((2506708038.8855705 / (32)) + (2112833754.216471 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.06382036116362423 = (3837955.6226314167 - 4082895.336597716) / 3837955.6226314167 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 14.80 μs 15.85 μs 1.07 0.01 False
SerializeObjectProperty - Duration of single invocation 16.29 μs 17.24 μs 1.06 0.01 False
SerializeToString - Duration of single invocation 10.65 μs 11.62 μs 1.09 0.01 False
SerializeToString - Duration of single invocation 7.00 μs 8.04 μs 1.15 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LoginViewModel>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.850550545131846 > 15.496619128185712.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -25.839290061143334 (T) = (0 -15999.364140659085) / Math.Sqrt((33568.091548023425 / (31)) + (25686.89991893764 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.08803342069325774 = (14704.846226566126 - 15999.364140659085) / 14704.846226566126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.23711697216097 > 17.014561235811623.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.53508904414575 (T) = (0 -17385.749231650465) / Math.Sqrt((32729.754151814286 / (32)) + (17674.038810083188 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08232398875559445 = (16063.350172659286 - 17385.749231650465) / 16063.350172659286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.62115752455691 > 10.973609629801215.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -34.322668770317335 (T) = (0 -11622.24541117066) / Math.Sqrt((17922.10738458226 / (32)) + (12673.529390512289 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11731878412541027 = (10401.90639976402 - 11622.24541117066) / 10401.90639976402 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LoginViewModel&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.044345483005367 > 7.347813359368967.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.86460305084507 (T) = (0 -7998.177039541712) / Math.Sqrt((31868.504675702465 / (32)) + (9432.156722974723 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.17037148030827823 = (6833.878964168689 - 7998.177039541712) / 6833.878964168689 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_ToString<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 15.86 μs 17.77 μs 1.12 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 21.28 μs 23.85 μs 1.12 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<Location>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.77383451013069 > 16.5762971474616.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.4538754987571 (T) = (0 -17638.33717779741) / Math.Sqrt((66861.31157461765 / (32)) + (10575.708657432175 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13464247794679254 = (15545.281902115192 - 17638.33717779741) / 15545.281902115192 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_ToString&lt;Location&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.849953298528714 > 22.376996689891726.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.58392291625533 (T) = (0 -23908.343207337643) / Math.Sqrt((68723.47870853523 / (32)) + (41605.17481570985 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12784144959583338 = (21198.319334606203 - 23908.343207337643) / 21198.319334606203 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 16.46 μs 18.49 μs 1.12 0.01 False
SerializeObjectProperty - Duration of single invocation 20.46 μs 22.41 μs 1.10 0.01 False
SerializeToString - Duration of single invocation 8.27 μs 9.59 μs 1.16 0.01 False
SerializeToString - Duration of single invocation 13.77 μs 15.12 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 18.49422480123675 > 17.284205601265825.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.534314625545232 (T) = (0 -18255.83046735234) / Math.Sqrt((24458.88555821421 / (31)) + (49593.12064356541 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.11058483490057144 = (16438.033271890257 - 18255.83046735234) / 16438.033271890257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.406868169642863 > 21.369077995892795.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.822289362982833 (T) = (0 -22402.679679531648) / Math.Sqrt((47218.56497413516 / (32)) + (44249.981700353484 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09780696419689282 = (20406.75675246828 - 22402.679679531648) / 20406.75675246828 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.592131810946805 > 8.698481016955242.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -40.89876620590795 (T) = (0 -9648.60429460009) / Math.Sqrt((11440.742173415349 / (32)) + (14119.670369660622 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.16719017410116202 = (8266.522892921332 - 9648.60429460009) / 8266.522892921332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.122776355275896 > 14.551889790014908.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.48816180562889 (T) = (0 -15469.313106588474) / Math.Sqrt((21352.58040658507 / (32)) + (36804.085697851086 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11437513729106658 = (13881.602872254321 - 15469.313106588474) / 13881.602872254321 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 2.21 ms 2.40 ms 1.08 0.02 False
SystemTextJson_SourceGen_ - Duration of single invocation 2.15 ms 2.41 ms 1.12 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToString<CollectionsOfPrimitives>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.3956704540702014 > 2.3173189023806735.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.82753380626185 (T) = (0 -2418876.3143194243) / Math.Sqrt((784579790.4072567 / (32)) + (483112215.6088419 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10051320022719006 = (2197953.0221173824 - 2418876.3143194243) / 2197953.0221173824 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_ToString&lt;CollectionsOfPrimitives&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 2.4093899162087915 > 2.265092370686999.
IsChangePoint: Marked as a change because one of 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -34.06198652576904 (T) = (0 -2382426.9399907035) / Math.Sqrt((872105278.8019714 / (32)) + (311272699.8828611 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10548981882367636 = (2155087.183458445 - 2382426.9399907035) / 2155087.183458445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.IO.Tests.Perf_Path

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
GetTempPath - Duration of single invocation 1.03 μs 1.28 μs 1.24 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.IO.Tests.Perf_Path*'

Payloads

Baseline
Compare

Histogram

System.IO.Tests.Perf_Path.GetTempPath

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2816476188505366 > 1.1235846894500108.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.125094979220037 (T) = (0 -1305.4417307678364) / Math.Sqrt((653.7453216002556 / (31)) + (1161.6696040215154 / (18))) is less than -2.011740513728388 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (18) - 2, .025) and -0.23804049401833707 = (1054.4418676732726 - 1305.4417307678364) / 1054.4418676732726 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Dec 27, 2022 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 476.24 μs 527.63 μs 1.11 0.01 False
DeserializeFromString - Duration of single invocation 435.86 μs 475.15 μs 1.09 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 393.17 μs 451.72 μs 1.15 0.01 False
DeserializeFromStream - Duration of single invocation 431.01 μs 491.85 μs 1.14 0.01 False
DeserializeFromReader - Duration of single invocation 477.37 μs 532.40 μs 1.12 0.01 False
DeserializeFromStream - Duration of single invocation 420.32 μs 479.10 μs 1.14 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 393.73 μs 451.32 μs 1.15 0.01 False
DeserializeFromString - Duration of single invocation 435.03 μs 478.80 μs 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>.DeserializeFromReader(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 527.630324107143 > 500.4896219970704.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.00841851210685 (T) = (0 -526848.9387999832) / Math.Sqrt((65082184.870495506 / (32)) + (17596529.08967578 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1056420838644398 = (476509.48393583304 - 526848.9387999832) / 476509.48393583304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 475.1518764568765 > 450.89229367788465.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.169666195838815 (T) = (0 -480317.8624776096) / Math.Sqrt((36563450.132136375 / (32)) + (17851769.853923313 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11978711771595113 = (428936.7638532243 - 480317.8624776096) / 428936.7638532243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 451.7206423406863 > 418.3552372756674.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.888135168465727 (T) = (0 -448899.9044336411) / Math.Sqrt((62751070.47420451 / (32)) + (27953519.92144712 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12648335738618963 = (398496.7034712665 - 448899.9044336411) / 398496.7034712665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 491.8516788194443 > 446.0192131117101.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -35.11676264151655 (T) = (0 -481861.57433928666) / Math.Sqrt((64371515.63369408 / (32)) + (13118331.924582068 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13719366928483093 = (423728.6817137527 - 481861.57433928666) / 423728.6817137527 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 532.4022069940477 > 505.42138069192526.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.603757045022025 (T) = (0 -529018.8299253214) / Math.Sqrt((63691922.38311607 / (32)) + (27172287.160859287 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.10817750171666651 = (477377.3417217221 - 529018.8299253214) / 477377.3417217221 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 479.10153602150535 > 442.7986775223215.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -33.501133253895475 (T) = (0 -481676.02661484043) / Math.Sqrt((45614468.49194723 / (32)) + (26051132.10123891 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13363690041613735 = (424894.44939382793 - 481676.02661484043) / 424894.44939382793 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 451.3237509803921 > 414.47225188527966.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.640804501284798 (T) = (0 -451630.14179477864) / Math.Sqrt((34032232.454117574 / (32)) + (33030155.103109114 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13542926594957305 = (397761.58263551095 - 451630.14179477864) / 397761.58263551095 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 478.8006735677084 > 450.1230516243133.
IsChangePoint: Marked as a change because one of 12/14/2022 5:20:21 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -33.08348158408239 (T) = (0 -479477.7263455286) / Math.Sqrt((56533048.144206725 / (32)) + (7243779.2242356585 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.11311540135837744 = (430752.9351946829 - 479477.7263455286) / 430752.9351946829 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeObjectProperty - Duration of single invocation 24.05 μs 26.29 μs 1.09 0.01 False
SerializeObjectProperty - Duration of single invocation 27.04 μs 30.04 μs 1.11 0.01 False
SerializeToString - Duration of single invocation 21.41 μs 23.76 μs 1.11 0.01 False
SerializeToString - Duration of single invocation 15.88 μs 17.92 μs 1.13 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Location&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Location>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.291388655021574 > 25.350302808580686.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -42.86585558907532 (T) = (0 -26356.66732554271) / Math.Sqrt((42640.3959506282 / (32)) + (29324.500053977426 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.09710366624961451 = (24023.862225929344 - 26356.66732554271) / 24023.862225929344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Location&gt;.SerializeObjectProperty(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.039272630494505 > 28.751197391850006.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -51.54195521533977 (T) = (0 -30041.30678502877) / Math.Sqrt((36519.568109009255 / (32)) + (31037.163573361544 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1011306753651634 = (27282.235848226002 - 30041.30678502877) / 27282.235848226002 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Location&gt;.SerializeToString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.763662506303582 > 22.255447742571256.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -51.23651056910486 (T) = (0 -23687.08650962816) / Math.Sqrt((58679.64249908364 / (32)) + (12274.59346507753 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.12169373615579475 = (21117.249518399916 - 23687.08650962816) / 21117.249518399916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.WriteJson&lt;Location&gt;.SerializeToString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.92416838877688 > 16.835493871891703.
IsChangePoint: Marked as a change because one of 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -39.56240897792171 (T) = (0 -17951.983800308743) / Math.Sqrt((53266.50437262478 / (32)) + (22079.633879721154 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13442743633789814 = (15824.70876961548 - 17951.983800308743) / 15824.70876961548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_DocumentParse

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 9.40 μs 10.30 μs 1.09 0.01 False
Parse - Duration of single invocation 9.79 μs 10.46 μs 1.07 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: HelloWorld)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.295445890560277 > 9.922802824029533.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -28.66352107995714 (T) = (0 -10165.61244303185) / Math.Sqrt((14777.401390071047 / (32)) + (5123.370083866757 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.08346493811454825 = (9382.502456168177 - 10165.61244303185) / 9382.502456168177 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.45874523269489 > 10.305128317218776.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 12/3/2022 2:55:34 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -29.682864953896495 (T) = (0 -10549.093339213645) / Math.Sqrt((14283.268048823244 / (32)) + (3642.636543661222 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.0771997405244927 = (9793.070813475364 - 10549.093339213645) / 9793.070813475364 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 279fb0436f475fbc35ffeff68330f970ee77831a
Compare 23059de6fbb664613c975131712559ccca50c702
Diff Diff

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Join_Enumerable - Duration of single invocation 3.66 μs 4.33 μs 1.18 0.03 False
ToLowerInvariant - Duration of single invocation 341.92 ns 367.80 ns 1.08 0.13 False
ToUpper - Duration of single invocation 887.14 ns 984.79 ns 1.11 0.06 False
ToUpper - Duration of single invocation 337.52 ns 432.91 ns 1.28 0.11 False
ToLowerInvariant - Duration of single invocation 316.36 ns 366.59 ns 1.16 0.11 False
ToUpperInvariant - Duration of single invocation 293.25 ns 383.52 ns 1.31 0.15 False
ToUpperInvariant - Duration of single invocation 290.02 ns 377.63 ns 1.30 0.12 False
ToUpperInvariant - Duration of single invocation 841.35 ns 923.39 ns 1.10 0.04 False
ToUpper - Duration of single invocation 336.93 ns 428.07 ns 1.27 0.13 False
ToLower - Duration of single invocation 359.71 ns 422.87 ns 1.18 0.14 False
ToLower - Duration of single invocation 358.95 ns 429.52 ns 1.20 0.11 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Join_Enumerable

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.325394897353833 > 3.8651999644678883.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -41.79729622954487 (T) = (0 -4401.488951137794) / Math.Sqrt((2582.289523718745 / (32)) + (3907.2661442333365 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1959794272343684 = (3680.2380132206595 - 4401.488951137794) / 3680.2380132206595 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 367.7987944674323 > 328.19394740684555.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.826449507990485 (T) = (0 -374.97595448243857) / Math.Sqrt((132.13743050458172 / (32)) + (64.91993985651914 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.20382381670038321 = (311.48740312367937 - 374.97595448243857) / 311.48740312367937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 984.790084758874 > 935.6914769290174.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -15.385153422882578 (T) = (0 -995.6957749412826) / Math.Sqrt((774.449657030452 / (32)) + (757.0033178706814 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1438694063668004 = (870.4628075540963 - 995.6957749412826) / 870.4628075540963 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 432.9118529040404 > 365.01976702497365.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.36825861544723 (T) = (0 -437.1704437612902) / Math.Sqrt((99.21195280631201 / (32)) + (190.45211789175377 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.2871590730202553 = (339.6397950530631 - 437.1704437612902) / 339.6397950530631 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 366.5888440629821 > 333.20215276361205.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.07221339908307 (T) = (0 -375.7431239154608) / Math.Sqrt((62.81191290669037 / (32)) + (59.182816255265976 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.2245821816887628 = (306.83373442302695 - 375.7431239154608) / 306.83373442302695 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 383.51503329725364 > 306.81716663731476.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -27.576358468383678 (T) = (0 -381.1992214069794) / Math.Sqrt((104.47114226173815 / (32)) + (131.52976345720057 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.3075543250181726 = (291.5360487233916 - 381.1992214069794) / 291.5360487233916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 377.6320256345815 > 311.9944143391511.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -30.447552094373584 (T) = (0 -382.87172234675995) / Math.Sqrt((114.15684651165017 / (32)) + (98.16881113245248 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.3138077357860589 = (291.42142485383187 - 382.87172234675995) / 291.42142485383187 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpperInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 923.3904516794169 > 882.7464561610244.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/17/2022 10:44:50 PM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -20.917118462823144 (T) = (0 -928.4745092867929) / Math.Sqrt((698.5690851091005 / (32)) + (123.97132703003496 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.13730370044498835 = (816.3822107705377 - 928.4745092867929) / 816.3822107705377 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToUpper(s: "test")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 428.0742515665515 > 360.62912295494556.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -31.666353854839492 (T) = (0 -432.565477690315) / Math.Sqrt((66.15084801788274 / (32)) + (113.2503614430211 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.26847409802291394 = (341.0124640026359 - 432.565477690315) / 341.0124640026359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "TEST")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 422.86792228482784 > 376.09930047263623.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -26.199507896985285 (T) = (0 -429.102200125989) / Math.Sqrt((94.34950764863306 / (32)) + (70.67366981353716 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1906017881872903 = (360.40782433168 - 429.102200125989) / 360.40782433168 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_String.ToLower(s: "TeSt")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 429.51795935958245 > 379.2633578693738.
IsChangePoint: Marked as a change because one of 12/8/2022 9:11:55 AM, 12/21/2022 9:23:17 AM, 12/26/2022 8:24:36 PM falls between 12/17/2022 10:44:50 PM and 12/26/2022 8:24:36 PM.
IsRegressionStdDev: Marked as regression because -22.69842636499406 (T) = (0 -428.84829225549845) / Math.Sqrt((116.29549256981916 / (32)) + (104.23280859539648 / (18))) is less than -2.010634757623041 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (18) - 2, .025) and -0.1940181835493017 = (359.16395425463054 - 428.84829225549845) / 359.16395425463054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

Could be dotnet/runtime@66e64e5 as regressions are related to System.Text.

This was referenced Dec 27, 2022
Closed
Closed
Closed
@lewing
Copy link
Member

lewing commented Jan 10, 2023 •
edited
Loading

this is dotnet/runtime#75012 and it does appear to be the cause cc @adamsitnik @SamMonoRT @BrzVlad

Could be dotnet/runtime@66e64e5 as regressions are related to System.Text.

@lewing lewing mentioned this issue Jan 10, 2023
Merged
@kotlarmilos
Copy link
Member

Mitigated by dotnet/runtime#81782. Closing this issue.

@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
@DrewScoggins
Copy link
Member

The Perf_String tests are still regressed by this change. Here is one as an example.

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory/refs/heads/main_x64_ubuntu%2018.04_LLVM=false_MonoAOT=true_MonoInterpreter=false_RunKind=micro_mono/System.Tests.Perf_String.Join_Enumerable.html

@kotlarmilos
Copy link
Member

This issue is reported for mono-interpreter where Vector128 intrinsics mitigated the Perf_String microbenchmarks. However, as you said it still exists in Mono AOT configuration. I wasn't able to find the mono-aot corresponding issue.

This was referenced Sep 14, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch-x64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@lewing @DrewScoggins @kotlarmilos