Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Windows/x86: 101 Regressions on 4/17/2023 1:30:24 AM #16587

Open
performanceautofiler bot opened this issue Apr 18, 2023 · 8 comments
Open

[Perf] Windows/x86: 101 Regressions on 4/17/2023 1:30:24 AM #16587

performanceautofiler bot opened this issue Apr 18, 2023 · 8 comments
Assignees
@EgorBo
Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 15.72 μs 20.32 μs 1.29 0.01 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<String, String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.321572388383185 > 16.438056454865176.
IsChangePoint: Marked as a change because one of 1/30/2023 9:42:46 PM, 3/14/2023 8:42:09 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -82.2200947281324 (T) = (0 -20252.25311416156) / Math.Sqrt((15150.697596127115 / (31)) + (15992.737637351174 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.2953487660486527 = (15634.594824943766 - 20252.25311416156) / 15634.594824943766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.DictionarySequentialKeys

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ContainsKey_17_Int_32ByteValue - Duration of single invocation 6.33 ns 9.53 ns 1.50 0.03 True 60.15482738373926 91.24106303644598 1.5167704239994177) Trace Trace
TryGetValue_3k_Int_Int - Duration of single invocation 6.27 ns 8.54 ns 1.36 0.02 True Trace Trace
ContainsValue_3k_Int_Int - Duration of single invocation 6.27 ns 8.72 ns 1.39 0.04 True 59.59259259259259 90.52994080145719 1.519147546077902) Trace Trace
TryGetValue_3k_Int_32ByteRefsValue - Duration of single invocation 6.47 ns 9.30 ns 1.44 0.09 True 59.4752899137001 90.42592592592592 1.5203948741928932) Trace Trace
ContainsValue_17_Int_Int - Duration of single invocation 6.40 ns 9.25 ns 1.45 0.08 True Trace Trace
TryGetValue_3k_Int_32ByteValue - Duration of single invocation 6.77 ns 8.60 ns 1.27 0.27 True 59.47414607803429 90.66564474173171 1.5244547542182778) Trace Trace
ContainsKey_3k_Int_32ByteRefsValue - Duration of single invocation 6.36 ns 8.74 ns 1.37 0.09 True Trace Trace
TryGetValue_17_Int_32ByteRefsValue - Duration of single invocation 6.35 ns 9.52 ns 1.50 0.00 True Trace Trace
ContainsKey_3k_Int_32ByteValue - Duration of single invocation 6.28 ns 9.06 ns 1.44 0.23 True 59.418175538103554 90.6656447417317 1.5258907551543692) Trace Trace
ContainsKey_17_Int_32ByteRefsValue - Duration of single invocation 6.29 ns 9.52 ns 1.51 0.01 True 60.27660558003935 91.2360489200543 1.5136228731212296) Trace Trace
TryGetValue_17_Int_32ByteValue - Duration of single invocation 6.34 ns 9.59 ns 1.51 0.01 True Trace Trace
TryGetValue_17_Int_Int - Duration of single invocation 6.30 ns 9.24 ns 1.47 0.07 True 59.71343426895397 91.20545768188387 1.527385902326223) Trace Trace

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.DictionarySequentialKeys*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.DictionarySequentialKeys.ContainsKey_17_Int_32ByteValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.528833612819078 > 6.6500342615410295.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -39.90319954062559 (T) = (0 -9.46463376470317) / Math.Sqrt((0.0953502474863154 / (30)) + (0.013512775787967612 / (5))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (5) - 2, .025) and -0.4777921963110393 = (6.404576900818262 - 9.46463376470317) / 6.404576900818262 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.538617119615454 > 6.5555447230538935.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -16.812087351926593 (T) = (0 -8.587337647625175) / Math.Sqrt((0.00016582441878742443 / (31)) + (0.11555262517713778 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3731172282456715 = (6.253899864468651 - 8.587337647625175) / 6.253899864468651 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.ContainsValue_3k_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.715908796296297 > 6.559512902821456.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -49.348604429370866 (T) = (0 -8.64184713945814) / Math.Sqrt((0.000649093000304845 / (31)) + (0.013907005820689211 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3815226438053994 = (6.255306185683791 - 8.64184713945814) / 6.255306185683791 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_32ByteRefsValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.298114202582289 > 6.739305830248543.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -20.88573329924112 (T) = (0 -9.029975745694014) / Math.Sqrt((0.014556840574712522 / (31)) + (0.09232160237508441 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4109360826403425 = (6.399989238914246 - 9.029975745694014) / 6.399989238914246 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.ContainsValue_17_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.253560773553312 > 6.717855115951034.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -49.87087599004371 (T) = (0 -9.250532641914528) / Math.Sqrt((0.09885703847252453 / (31)) + (0.0001911898661813185 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4408356318213421 = (6.420255327959267 - 9.250532641914528) / 6.420255327959267 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_3k_Int_32ByteValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.598611509758898 > 6.892882312802064.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -12.363034896918505 (T) = (0 -9.241134579600102) / Math.Sqrt((0.042656827847029476 / (31)) + (0.28898012682394125 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4240267236930047 = (6.489439015325901 - 9.241134579600102) / 6.489439015325901 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.ContainsKey_3k_Int_32ByteRefsValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.740728423945335 > 6.621385055558496.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -24.271248289374658 (T) = (0 -8.764346611138135) / Math.Sqrt((0.015016618388972591 / (31)) + (0.0533607723911861 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3664542323482042 = (6.413933524928171 - 8.764346611138135) / 6.413933524928171 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_32ByteRefsValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.524942109734958 > 6.68262228784433.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -69.39157265662564 (T) = (0 -9.478216099725463) / Math.Sqrt((0.0019418974162602888 / (31)) + (0.01186256045804406 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.49398415261931433 = (6.344254778812657 - 9.478216099725463) / 6.344254778812657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.ContainsKey_3k_Int_32ByteValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.064379633443163 > 6.593569530415318.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -14.487068972957594 (T) = (0 -9.076754260000628) / Math.Sqrt((0.029131844984366117 / (31)) + (0.18980163483419105 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.40461033812467984 = (6.462115516050639 - 9.076754260000628) / 6.462115516050639 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.ContainsKey_17_Int_32ByteRefsValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.524458650961726 > 6.606140174044635.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -35.03083207913985 (T) = (0 -9.637055966398474) / Math.Sqrt((0.002137635354863422 / (31)) + (0.05247569931719308 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.518103077711735 = (6.348090658589921 - 9.637055966398474) / 6.348090658589921 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_32ByteValue

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.591220868986593 > 6.696748568806835.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -72.3065085170644 (T) = (0 -9.49208948865806) / Math.Sqrt((0.0010504663128280982 / (31)) + (0.011188180824427699 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4968214499850339 = (6.341497503762367 - 9.49208948865806) / 6.341497503762367 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.Tests.DictionarySequentialKeys.TryGetValue_17_Int_Int

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.240513147356994 > 6.728378593382796.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -108.87480008911008 (T) = (0 -9.245072543423893) / Math.Sqrt((0.021381041312389305 / (31)) + (2.2039377614003065E-05 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4494888292742804 = (6.378160601660276 - 9.245072543423893) / 6.378160601660276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 344.37 ns 383.71 ns 1.11 0.04 False
SerializeObjectProperty - Duration of single invocation 742.05 ns 796.25 ns 1.07 0.05 False
SerializeToWriter - Duration of single invocation 406.96 ns 451.54 ns 1.11 0.07 False
SerializeToUtf8Bytes - Duration of single invocation 484.13 ns 529.32 ns 1.09 0.08 False
SerializeToStream - Duration of single invocation 598.97 ns 644.36 ns 1.08 0.20 False
SerializeToString - Duration of single invocation 404.49 ns 429.14 ns 1.06 0.06 False
SerializeToWriter - Duration of single invocation 271.14 ns 302.81 ns 1.12 0.01 False

graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;ClassRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 383.7068189513499 > 361.57290847786834.
IsChangePoint: Marked as a change because one of 3/20/2023 11:25:43 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -7.389417588249708 (T) = (0 -392.2676557185635) / Math.Sqrt((120.98164961731231 / (31)) + (141.10008720228288 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.10943545604355241 = (353.57411157334104 - 392.2676557185635) / 353.57411157334104 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeObjectProperty(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 796.2484580418399 > 780.409015029806.
IsChangePoint: Marked as a change because one of 3/7/2023 10:25:17 PM, 3/15/2023 9:07:40 PM, 4/12/2023 4:44:56 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -12.679684227480408 (T) = (0 -785.9823739773675) / Math.Sqrt((172.574649727789 / (30)) + (82.78655008822287 / (7))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (7) - 2, .025) and -0.07254544085250378 = (732.8196494431378 - 785.9823739773675) / 732.8196494431378 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 451.54412767034717 > 426.55583231683534.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -30.296838198750354 (T) = (0 -451.8912921256838) / Math.Sqrt((34.5371569857774 / (31)) + (8.080284611131244 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11753403901645232 = (404.364678254808 - 451.8912921256838) / 404.364678254808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 529.3162363978765 > 506.43328392123414.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -10.85205292772303 (T) = (0 -534.4381911201689) / Math.Sqrt((130.54835701093575 / (31)) + (93.23082441146252 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09918990871673865 = (486.2109694439468 - 534.4381911201689) / 486.2109694439468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToStream(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 644.3635361231221 > 620.497590243585.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -10.30293830536127 (T) = (0 -657.0900168108759) / Math.Sqrt((262.43883483046403 / (31)) + (144.495714487435 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09824245347069063 = (598.310523085613 - 657.0900168108759) / 598.310523085613 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 429.13842823935227 > 419.113351779623.
IsChangePoint: Marked as a change because one of 2/13/2023 6:54:14 PM, 3/15/2023 5:33:41 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -10.378533578488591 (T) = (0 -432.8522429053689) / Math.Sqrt((48.37453197808898 / (31)) + (51.69716854740558 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08282414505752497 = (399.7438040896046 - 432.8522429053689) / 399.7438040896046 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<ClassRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 302.8080740246923 > 286.21149970265265.
IsChangePoint: Marked as a change because one of 3/15/2023 5:33:41 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -15.422858049642706 (T) = (0 -307.11786465794626) / Math.Sqrt((20.717964226211606 / (31)) + (20.605307005073634 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11323277155540142 = (275.87928823622684 - 307.11786465794626) / 275.87928823622684 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 3.95 μs 7.73 μs 1.95 0.08 True 36612.54815630159 52264.913185490805 1.4275136754308426) Trace Trace
IDictionary - Duration of single invocation 6.82 μs 8.51 μs 1.25 0.48 False Trace Trace
FrozenDictionary - Duration of single invocation 6.34 μs 9.16 μs 1.44 0.03 True 42081.00991325899 57946.53349001175 1.3770233558903684) Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.72675771754983 > 4.131169703129867.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -21.12742437938295 (T) = (0 -8000.859106540946) / Math.Sqrt((92782.46288005571 / (31)) + (179854.40041989583 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.9211168694269524 = (4164.691505169862 - 8000.859106540946) / 4164.691505169862 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.TryGetValueTrue<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.50925058972963 > 7.329739353957377.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -16.585809256751453 (T) = (0 -8309.328055276403) / Math.Sqrt((181196.42792702996 / (31)) + (15894.789814791753 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.22543523464214033 = (6780.715798255097 - 8309.328055276403) / 6780.715798255097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.TryGetValueTrue<Int32, Int32>.FrozenDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.164627116170461 > 6.670117349151122.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -49.1835335630277 (T) = (0 -9227.506173077782) / Math.Sqrt((45830.72654753914 / (31)) + (11186.722603653736 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.44544831968863535 = (6383.836798167563 - 9227.506173077782) / 6383.836798167563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 961.94 ns 1.07 μs 1.11 0.06 False 9232.467877265659 10090.432847629238 1.0929291043054978) Trace Trace
WriteBasicUtf8 - Duration of single invocation 2.13 ms 3.17 ms 1.49 0.00 True 19491737.891737893 26894017.094017092 1.3797649672591206) Trace Trace
WriteBasicUtf16 - Duration of single invocation 3.05 ms 3.94 ms 1.29 0.00 True Trace Trace
WriteBasicUtf16 - Duration of single invocation 2.22 ms 3.22 ms 1.45 0.00 True Trace Trace
WriteBasicUtf8 - Duration of single invocation 3.13 ms 4.01 ms 1.28 0.00 True Trace Trace
WriteBasicUtf16 - Duration of single invocation 1.19 μs 1.34 μs 1.13 0.03 False Trace Trace
WriteBasicUtf8 - Duration of single invocation 2.23 ms 3.21 ms 1.44 0.00 True Trace Trace
WriteBasicUtf8 - Duration of single invocation 3.08 ms 3.96 ms 1.28 0.00 True 28851028.806584366 35717460.31746032 1.2379960713674412) Trace Trace
WriteBasicUtf16 - Duration of single invocation 1.24 μs 1.35 μs 1.09 0.01 False 11518.582971804864 12315.936349602185 1.0692232177993664) Trace Trace
WriteBasicUtf8 - Duration of single invocation 831.30 ns 956.50 ns 1.15 0.02 False Trace Trace
WriteBasicUtf8 - Duration of single invocation 1.05 μs 1.16 μs 1.10 0.01 True Trace Trace
WriteBasicUtf8 - Duration of single invocation 1.09 μs 1.22 μs 1.12 0.01 False 10345.777836099993 11171.158778040895 1.0797794960433873) Trace Trace
WriteBasicUtf16 - Duration of single invocation 924.93 ns 1.03 μs 1.11 0.06 False 8760.641507329268 9608.580489336924 1.0967895994029957) Trace Trace
WriteBasicUtf16 - Duration of single invocation 3.13 ms 4.02 ms 1.28 0.00 True 29832911.392405063 36720430.107526876 1.2308698143646568) Trace Trace
WriteBasicUtf16 - Duration of single invocation 2.13 ms 3.16 ms 1.48 0.00 True Trace Trace

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0689434164655403 > 1016.9509447720881.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -8.56509141023933 (T) = (0 -1079.5530652541104) / Math.Sqrt((487.6499865202644 / (30)) + (780.8198697502386 / (6))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (6) - 2, .025) and -0.10618790640120575 = (975.9219559416923 - 1079.5530652541104) / 975.9219559416923 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.1674958649789025 > 2.2359389626249455.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -220.73770811951883 (T) = (0 -3167457.541691782) / Math.Sqrt((197744036.55154717 / (31)) + (97711910.03814523 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4964880106747643 = (2116593.998146086 - 3167457.541691782) / 2116593.998146086 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.9442841269841264 > 3.2021809024390246.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -125.79004166607947 (T) = (0 -3950464.247339962) / Math.Sqrt((1285292813.7180092 / (31)) + (79406024.26016822 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3080283293757242 = (3020167.1925755455 - 3950464.247339962) / 3020167.1925755455 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2180078632478635 > 2.326886586283186.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -139.16990088397782 (T) = (0 -3220832.3294674396) / Math.Sqrt((459114895.2784062 / (31)) + (228124557.271695 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.45786675298227597 = (2209277.5782689084 - 3220832.3294674396) / 2209277.5782689084 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.009704516129032 > 3.298309282832278.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -113.10843239897747 (T) = (0 -4024172.765567765) / Math.Sqrt((968442590.7884991 / (31)) + (207170081.89102203 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.2952418987205698 = (3106888.967645976 - 4024172.765567765) / 3106888.967645976 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3417980601705355 > 1.2476823878042511.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -19.135510488353066 (T) = (0 -1309.5893028904231) / Math.Sqrt((155.01568697591887 / (31)) + (254.5358048546662 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11188158923692698 = (1177.8136409194265 - 1309.5893028904231) / 1177.8136409194265 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.2134265567765574 > 2.3367738102678577.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -157.90979988352336 (T) = (0 -3231311.810358953) / Math.Sqrt((194615860.6133991 / (31)) + (215946327.33324385 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4656685323409695 = (2204667.521378721 - 3231311.810358953) / 2204667.521378721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.957947936507936 > 3.239373696581197.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -78.52631009270826 (T) = (0 -3957530.546737213) / Math.Sqrt((3711164563.637673 / (31)) + (102019717.52489331 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3020980156820426 = (3039349.188059585 - 3957530.546737213) / 3039349.188059585 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.3478807414616134 > 1.3031077151423573.
IsChangePoint: Marked as a change because one of 2/6/2023 2:52:08 AM, 2/9/2023 4:27:10 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -32.079639924582004 (T) = (0 -1352.324362349391) / Math.Sqrt((341.647970200693 / (31)) + (44.81550609954052 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11359119168160754 = (1214.381338907035 - 1352.324362349391) / 1214.381338907035 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 956.495115203141 > 869.2694728556717.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -33.75933887948399 (T) = (0 -944.718348967223) / Math.Sqrt((105.63061828674655 / (30)) + (47.48395675893802 / (6))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (6) - 2, .025) and -0.13744909726814747 = (830.558792684602 - 944.718348967223) / 830.558792684602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.1562366202388374 > 1.0955524105383472.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -34.905318510777626 (T) = (0 -1155.8190065053693) / Math.Sqrt((219.29328722544025 / (31)) + (24.58211835995737 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11226834292582978 = (1039.1548171414995 - 1155.8190065053693) / 1039.1548171414995 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2186326849967024 > 1.1361647580026442.
IsChangePoint: Marked as a change because one of 2/8/2023 7:09:09 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -33.706580857517636 (T) = (0 -1211.9167288750634) / Math.Sqrt((97.7305792351099 / (31)) + (66.15262165945452 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11697510855895203 = (1084.9988684515977 - 1211.9167288750634) / 1084.9988684515977 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.0286231257386043 > 970.4638869239994.
IsChangePoint: Marked as a change because one of 2/9/2023 10:24:28 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -23.459366064127448 (T) = (0 -1032.9511514329001) / Math.Sqrt((104.29042082288038 / (31)) + (117.1490002006062 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.12190022803134883 = (920.7156979061036 - 1032.9511514329001) / 920.7156979061036 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.020720752688172 > 3.285538025.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -130.07591651771722 (T) = (0 -4020732.031686952) / Math.Sqrt((1418090852.9358206 / (31)) + (22715483.93841756 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.2948040086820734 = (3105282.3475419157 - 4020732.031686952) / 3105282.3475419157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.162356864654333 > 2.234185452991453.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -279.47125660600165 (T) = (0 -3167451.658336975) / Math.Sqrt((292018352.4429912 / (30)) + (26445759.340248577 / (6))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (6) - 2, .025) and -0.4965572486790376 = (2116492.142971999 - 3167451.658336975) / 2116492.142971999 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<String, String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 14.94 μs 19.87 μs 1.33 0.23 False 127266.43880208334 142747.4245768948 1.1216423270779698) Trace Trace
IDictionary - Duration of single invocation 19.59 μs 21.17 μs 1.08 0.22 False 137577.83882783883 152604.1666666667 1.109220554464672) Trace Trace
FrozenDictionary - Duration of single invocation 16.74 μs 21.25 μs 1.27 0.32 False Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<String, String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.868153781157584 > 15.746738470416568.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -25.851574070998264 (T) = (0 -19767.102327123754) / Math.Sqrt((776787.9746234965 / (31)) + (19221.56228610658 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.28181588592292817 = (15421.17128068757 - 19767.102327123754) / 15421.17128068757 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.TryGetValueTrue<String, String>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.16958402076864 > 17.84872807077007.
IsChangePoint: Marked as a change because one of 1/30/2023 9:42:46 PM, 3/14/2023 8:42:09 PM, 4/12/2023 4:44:56 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -4.996903448418449 (T) = (0 -20203.611741122477) / Math.Sqrt((1153546.3890273226 / (30)) + (1779956.2842603999 / (7))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (7) - 2, .025) and -0.15448840490615595 = (17500.05600339031 - 20203.611741122477) / 17500.05600339031 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.TryGetValueTrue<String, String>.FrozenDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 21.245972035040435 > 17.527448078443175.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -21.899911546921963 (T) = (0 -21152.56135612321) / Math.Sqrt((981123.976714094 / (31)) + (16460.068621143007 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.23763058497713094 = (17091.175357882796 - 21152.56135612321) / 17091.175357882796 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IntegerFormatting - Duration of single invocation 1.25 ms 1.42 ms 1.14 0.03 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.LowLevelPerf.IntegerFormatting

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4204625347222224 > 1.3057933039772727.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -136.40407084852802 (T) = (0 -1425016.057964027) / Math.Sqrt((13480762.123331508 / (31)) + (7829842.149736476 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.14450373295813565 = (1245095.1595246147 - 1425016.057964027) / 1245095.1595246147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf8 - Duration of single invocation 6.74 ms 7.64 ms 1.13 0.00 False Trace Trace
WriteDeepUtf8 - Duration of single invocation 36.45 ms 39.36 ms 1.08 0.08 False 72828571.42857143 80288888.88888888 1.1024366854104004) Trace Trace
WriteDeepUtf8 - Duration of single invocation 6.68 ms 7.66 ms 1.15 0.01 False Trace Trace
WriteDeepUtf16 - Duration of single invocation 7.24 ms 7.73 ms 1.07 0.01 False 59846296.29629629 67225806.4516129 1.123307716801404) Trace Trace

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.636095959595959 > 7.066234047297297.
IsChangePoint: Marked as a change because one of 2/20/2023 2:42:44 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -11.592599857893811 (T) = (0 -7775224.406445406) / Math.Sqrt((10323798917.174181 / (31)) + (36083099554.84495 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.13479202035163898 = (6851673.493470716 - 7775224.406445406) / 6851673.493470716 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: True, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.35543333333334 > 38.51005734890111.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -5.564635756064362 (T) = (0 -39552562.84391534) / Math.Sqrt((2294613608397.317 / (31)) + (800807883363.0964 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.0684734321135081 = (37017825.29648666 - 39552562.84391534) / 37017825.29648666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.661497916666667 > 6.966993802631579.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -32.77932277865133 (T) = (0 -7686409.176961926) / Math.Sqrt((5037723783.604642 / (31)) + (6200471666.403733 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.17299032724838392 = (6552832.532722418 - 7686409.176961926) / 6552832.532722418 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.729700952380952 > 7.585738100000002.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 3/6/2023 11:14:52 AM, 3/14/2023 8:42:09 PM, 4/8/2023 3:09:32 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -22.77762981722057 (T) = (0 -7783458.823272792) / Math.Sqrt((37842137399.83965 / (31)) + (1791670592.3080335 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1287539791158179 = (6895620.274463861 - 7783458.823272792) / 6895620.274463861 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndRemove<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 16.95 μs 19.07 μs 1.13 0.01 False Trace Trace
Dictionary - Duration of single invocation 21.23 μs 28.64 μs 1.35 0.08 True 133920.8633093525 190870.69525666017 1.4252498866868528) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<Int32>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.065715733982156 > 17.888716421715838.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.874044983444243 (T) = (0 -18805.00498609983) / Math.Sqrt((53180.411940379745 / (31)) + (36613.15726850955 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09738443586403854 = (17136.20530009931 - 18805.00498609983) / 17136.20530009931 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.CreateAddAndRemove<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.64223982862376 > 22.12027637768817.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -54.398132455442216 (T) = (0 -28598.37536733874) / Math.Sqrt((221177.59847370823 / (31)) + (64089.94236962071 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.3402925402393312 = (21337.412921981966 - 28598.37536733874) / 21337.412921981966 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 6.44 μs 9.00 μs 1.40 0.09 False 35882.944865674435 51964.01298205422 1.4481535218633326) Trace Trace
IDictionary - Duration of single invocation 7.52 μs 8.55 μs 1.14 0.54 False 36742.95500544917 52127.46191007061 1.418706304442303) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyFalse&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyFalse<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.995450636640058 > 6.758398341172543.
IsChangePoint: Marked as a change because one of 3/25/2023 10:49:06 AM, 4/10/2023 1:33:42 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.841908977423017 (T) = (0 -8716.72094069751) / Math.Sqrt((853659.1782304317 / (31)) + (105073.93046345026 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.5083997355645802 = (5778.787104755705 - 8716.72094069751) / 5778.787104755705 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.ContainsKeyFalse<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.547059633441675 > 7.91367508125079.
IsChangePoint: Marked as a change because one of 2/20/2023 10:31:48 AM, 4/12/2023 4:47:47 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -8.539205151334958 (T) = (0 -8901.102911074084) / Math.Sqrt((261581.3554577877 / (27)) + (62168.07019869867 / (9))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (27) + (9) - 2, .025) and -0.14101435625032524 = (7801.043748761813 - 8901.102911074084) / 7801.043748761813 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToString - Duration of single invocation 383.04 ns 429.59 ns 1.12 0.04 False
SerializeToWriter - Duration of single invocation 406.06 ns 449.87 ns 1.11 0.10 False
SerializeToUtf8Bytes - Duration of single invocation 481.57 ns 537.23 ns 1.12 0.08 False
SerializeToStream - Duration of single invocation 573.21 ns 649.00 ns 1.13 0.09 False
SerializeToStream - Duration of single invocation 409.03 ns 457.00 ns 1.12 0.05 False
SerializeToWriter - Duration of single invocation 274.04 ns 301.15 ns 1.10 0.01 False
SerializeToUtf8Bytes - Duration of single invocation 345.33 ns 393.85 ns 1.14 0.04 False
SerializeToString - Duration of single invocation 523.98 ns 580.96 ns 1.11 0.10 False

graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;StructRecord&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 429.5898982469568 > 406.9538963607324.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -8.573491241060717 (T) = (0 -431.3326533480167) / Math.Sqrt((83.81308104077085 / (31)) + (103.1260134596874 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09727288206415895 = (393.0951547226847 - 431.3326533480167) / 393.0951547226847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 449.8668918454852 > 424.6778169725521.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -30.04746960791267 (T) = (0 -448.2370553749835) / Math.Sqrt((12.779717251387257 / (31)) + (10.949092410628678 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.11143644681287741 = (403.2952641244895 - 448.2370553749835) / 403.2952641244895 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 537.2256436319427 > 503.81010191158384.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -11.572888148752144 (T) = (0 -523.4633880518543) / Math.Sqrt((52.774391248369035 / (31)) + (67.16513461236949 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08624213852081532 = (481.9030393763566 - 523.4633880518543) / 481.9030393763566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToStream(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 649.0001337063397 > 601.1548456467176.
IsChangePoint: Marked as a change because one of 4/12/2023 4:44:56 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.058079472145895 (T) = (0 -634.2624020715804) / Math.Sqrt((161.78048890659707 / (30)) + (90.61860693060551 / (7))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (7) - 2, .025) and -0.09668780115375386 = (578.3436283364456 - 634.2624020715804) / 578.3436283364456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToStream(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 456.9950898134076 > 434.54482712542153.
IsChangePoint: Marked as a change because one of 2/24/2023 6:56:27 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -8.641461973573245 (T) = (0 -452.49515688026685) / Math.Sqrt((40.55197092433344 / (31)) + (96.6279050882662 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08659114724623225 = (416.4355268557392 - 452.49515688026685) / 416.4355268557392 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 301.152355005601 > 286.219569691537.
IsChangePoint: Marked as a change because one of 3/21/2023 3:33:39 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -41.8021988577596 (T) = (0 -300.9215328261509) / Math.Sqrt((12.640281985491864 / (31)) + (1.8460151423067857 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.13314051597779525 = (265.56418077283513 - 300.9215328261509) / 265.56418077283513 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 393.84551856213676 > 359.2939047946899.
IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.507193491878807 (T) = (0 -388.2822651601671) / Math.Sqrt((53.94208589098896 / (31)) + (33.453036393372365 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.10386196126060136 = (351.74893128553134 - 388.2822651601671) / 351.74893128553134 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Text.Json.Serialization.Tests.WriteJson<StructRecord>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 580.9615265434624 > 549.6370538120485.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -10.203264485041128 (T) = (0 -573.5374947510635) / Math.Sqrt((133.85425368697625 / (31)) + (85.87798128317881 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08317495555586452 = (529.4966356166674 - 573.5374947510635) / 529.4966356166674 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToStream - Duration of single invocation 914.09 ns 995.28 ns 1.09 0.01 False 6838.873209992683 7282.1012530003745 1.0648100980085529) Trace Trace
SerializeToString - Duration of single invocation 868.09 ns 961.59 ns 1.11 0.01 False Trace Trace
SerializeToUtf8Bytes - Duration of single invocation 824.33 ns 902.84 ns 1.10 0.01 False 6245.128234377079 6739.690721649485 1.0791917265285322) Trace Trace
SerializeToString - Duration of single invocation 517.55 ns 623.47 ns 1.20 0.02 False 4793.07483586501 5303.111168040078 1.1064110930124924) Trace Trace
SerializeObjectProperty - Duration of single invocation 804.28 ns 882.16 ns 1.10 0.04 False 6564.921905640267 7108.133744368034 1.0827446002459016) Trace Trace
SerializeToWriter - Duration of single invocation 398.36 ns 487.28 ns 1.22 0.02 True 3671.410392915574 4148.295460384999 1.1298915175458544) Trace Trace
SerializeToWriter - Duration of single invocation 754.57 ns 796.92 ns 1.06 0.03 False Trace Trace
SerializeToUtf8Bytes - Duration of single invocation 472.58 ns 566.10 ns 1.20 0.05 False Trace Trace
SerializeToStream - Duration of single invocation 549.01 ns 632.07 ns 1.15 0.05 False 4887.687969924812 5430.924374740093 1.1111438390007613) Trace Trace

graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;LargeStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToStream(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 995.2756500880689 > 962.5330099443137.
IsChangePoint: Marked as a change because one of 4/5/2023 8:56:55 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -15.54630076039353 (T) = (0 -985.4543014192018) / Math.Sqrt((87.51330680065357 / (31)) + (86.43724971843591 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.07007059257800763 = (920.9245710089567 - 985.4543014192018) / 920.9245710089567 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToString(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 961.5944534011286 > 911.6105780421809.
IsChangePoint: Marked as a change because one of 2/13/2023 12:37:33 PM, 3/14/2023 10:43:28 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -14.785546510365064 (T) = (0 -947.5342507190002) / Math.Sqrt((111.71133453452418 / (31)) + (125.86417577545134 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08385166299738872 = (874.22871880696 - 947.5342507190002) / 874.22871880696 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 902.8410099139762 > 868.2146653011838.
IsChangePoint: Marked as a change because one of 2/14/2023 2:39:38 AM, 2/24/2023 12:21:41 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -10.453784105260725 (T) = (0 -891.346240676501) / Math.Sqrt((465.17349155024084 / (31)) + (108.5906938865202 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.0723617281978306 = (831.1992280575537 - 891.346240676501) / 831.1992280575537 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 623.4661552953842 > 545.1874986768838.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -22.498999221435916 (T) = (0 -629.6639255253763) / Math.Sqrt((50.835277468874324 / (31)) + (115.40688907189697 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.19510409587783503 = (526.86952349776 - 629.6639255253763) / 526.86952349776 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeObjectProperty(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 882.162133476735 > 848.4304405911198.
IsChangePoint: Marked as a change because one of 3/17/2023 9:25:32 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.643033240441522 (T) = (0 -882.4453412545126) / Math.Sqrt((221.86477658073264 / (31)) + (143.51208091150016 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09431369910564019 = (806.3915694153485 - 882.4453412545126) / 806.3915694153485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 487.28187249963736 > 421.6906943763206.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -37.00121157861019 (T) = (0 -481.3292020467473) / Math.Sqrt((68.76772118115123 / (31)) + (12.807060682020722 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1910197309584388 = (404.13201354725794 - 481.3292020467473) / 404.13201354725794 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 796.9154531486072 > 791.8782054936552.
IsChangePoint: Marked as a change because one of 2/8/2023 3:55:57 PM, 2/21/2023 5:46:41 AM, 3/27/2023 3:21:55 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -22.68599957814102 (T) = (0 -802.3524382607279) / Math.Sqrt((55.577442676771675 / (31)) + (13.492576012059525 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.06026761598419669 = (756.7452086291835 - 802.3524382607279) / 756.7452086291835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 566.1015279991637 > 498.2721985682903.
IsChangePoint: Marked as a change because one of 2/20/2023 2:42:44 PM, 3/21/2023 7:06:30 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.60693719433438 (T) = (0 -571.7845643546934) / Math.Sqrt((32.571451930701535 / (31)) + (138.30209554557266 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.19013214777571572 = (480.4378786198859 - 571.7845643546934) / 480.4378786198859 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<LargeStructWithProperties>.SerializeToStream(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 632.065541593358 > 574.0744751308443.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.19232894821296 (T) = (0 -635.5004571376345) / Math.Sqrt((72.46778109622655 / (31)) + (111.27494621515227 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1505096265828863 = (552.3643109576807 - 635.5004571376345) / 552.3643109576807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToUtf8Bytes - Duration of single invocation 190.19 ns 212.82 ns 1.12 0.10 False Trace Trace
SerializeToWriter - Duration of single invocation 187.22 ns 202.87 ns 1.08 0.04 False 1722.963652665701 1807.948254789892 1.049324663345455) Trace Trace
SerializeToStream - Duration of single invocation 244.30 ns 282.86 ns 1.16 0.22 False 2361.213095450727 2433.055604336995 1.0304261013225298) Trace Trace
SerializeToString - Duration of single invocation 204.99 ns 237.83 ns 1.16 0.10 False 2111.9509211751206 2214.8308122552244 1.0487132016414755) Trace Trace
SerializeToWriter - Duration of single invocation 117.10 ns 132.86 ns 1.13 0.05 False 1150.023252697707 1249.6453900709218 1.0866261939830544) Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;SimpleStructWithProperties&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 212.81624604584056 > 200.37321056314138.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -12.188091510527425 (T) = (0 -211.86773328739045) / Math.Sqrt((20.303657862682545 / (31)) + (13.868164103770065 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.10997469688273884 = (190.87618292777427 - 211.86773328739045) / 190.87618292777427 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 202.869497705008 > 198.41821307049392.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -5.72951528257086 (T) = (0 -205.0911229062032) / Math.Sqrt((8.683035272390503 / (31)) + (32.655254528453185 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.07161571262990327 = (191.38495310308514 - 205.0911229062032) / 191.38495310308514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToStream(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 282.8592058088581 > 261.87175970863353.
IsChangePoint: Marked as a change because one of 4/17/2023 7:48:00 AM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -22.375173673137173 (T) = (0 -281.6844267553335) / Math.Sqrt((41.16828931649921 / (32)) + (1.182380865496004 / (3))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (3) - 2, .025) and -0.11479866813236184 = (252.67739799801112 - 281.6844267553335) / 252.67739799801112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToString(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 237.83265197156612 > 221.88971267716403.
IsChangePoint: Marked as a change because one of 3/24/2023 7:28:10 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -9.090118181419827 (T) = (0 -230.1937193646456) / Math.Sqrt((19.949841115893005 / (31)) + (17.308530717846573 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08011796341343827 = (213.11905473470404 - 230.1937193646456) / 213.11905473470404 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<SimpleStructWithProperties>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 132.85995491879208 > 122.87318289933155.
IsChangePoint: Marked as a change because one of 2/15/2023 8:55:14 PM, 3/21/2023 2:16:57 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -41.01980595704394 (T) = (0 -133.86687940174872) / Math.Sqrt((0.8980300049236349 / (31)) + (0.7987572161549114 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1407304915411687 = (117.3518902093076 - 133.86687940174872) / 117.3518902093076 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IDictionary - Duration of single invocation 8.53 μs 11.86 μs 1.39 0.13 True 53565.192083818394 71540.68984101321 1.335581691354096) Trace Trace
Dictionary - Duration of single invocation 8.06 μs 11.64 μs 1.45 0.14 False 49904.34863755132 67881.00745309689 1.36022229136999) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.86167330568591 > 8.909047392478337.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -86.36380877252749 (T) = (0 -11946.672719398002) / Math.Sqrt((18601.833087969444 / (31)) + (5957.079394653214 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4055260440003324 = (8499.787514001539 - 11946.672719398002) / 8499.787514001539 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.AddGivenSize<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.642641579731746 > 8.431807442881448.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/16/2023 6:27:39 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -87.88135063366542 (T) = (0 -11636.088422113035) / Math.Sqrt((42377.017073932 / (31)) + (815.9442749373903 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.41402292430316934 = (8229.066320015498 - 11636.088422113035) / 8229.066320015498 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 58.94 ns 63.08 ns 1.07 0.00 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 63.079332441040414 > 61.28638466043339.
IsChangePoint: Marked as a change because one of 2/19/2023 8:14:07 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.985846959442377 (T) = (0 -63.34170496184618) / Math.Sqrt((0.12139313014318127 / (31)) + (0.37007011270598833 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08314989510790716 = (58.479168255411096 - 63.34170496184618) / 58.479168255411096 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString_Format_Flags_Large - Duration of single invocation 17.93 ns 19.85 ns 1.11 0.07 False 200.78644016535986 207.62224901499957 1.0340451717955157) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.84734388513028 > 18.871787750587444.
IsChangePoint: Marked as a change because one of 3/16/2023 1:49:47 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -20.396062093668764 (T) = (0 -19.69540417430501) / Math.Sqrt((0.10218948406711555 / (30)) + (0.01085847976306652 / (6))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (6) - 2, .025) and -0.08083757292564452 = (18.222353356011563 - 19.69540417430501) / 18.222353356011563 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 48.41 ns 50.97 ns 1.05 0.01 False 355.9467629491424 392.7137192418501 1.1032934138467243) Trace Trace
TryFormat - Duration of single invocation 9.01 ns 10.49 ns 1.16 0.02 False 108.07739318996774 123.04348732968342 1.1384757135417742) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.TryFormat(value: 18446744073709551615)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.96557944790632 > 50.92688641732697.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -11.721719615723005 (T) = (0 -51.28272159987896) / Math.Sqrt((0.26366010672426127 / (31)) + (0.28608671295067845 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.05728312710965271 = (48.504246672386685 - 51.28272159987896) / 48.504246672386685 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Tests.Perf_UInt64.TryFormat(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.4899384014869 > 9.477040300242297.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -47.58390155373436 (T) = (0 -10.527342994925776) / Math.Sqrt((0.013153520322238344 / (31)) + (0.003268061667236885 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1637403360521352 = (9.046127103094719 - 10.527342994925776) / 9.046127103094719 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterInt64 - Duration of single invocation 12.09 ns 17.76 ns 1.47 0.05 False Trace Trace
FormatterUInt64 - Duration of single invocation 2.41 ns 14.87 ns 6.17 0.12 True Trace Trace
FormatterDecimal - Duration of single invocation 50.19 ns 85.76 ns 1.71 0.02 True 519.5154963421676 799.1068109153096 1.5381770448460217) Trace Trace
FormatterInt32 - Duration of single invocation 1.61 ns 10.03 ns 6.23 0.46 True 32.069214587077774 66.82548873930115 2.083789378684327) Trace Trace
FormatterUInt32 - Duration of single invocation 1.61 ns 9.59 ns 5.97 0.08 True Trace Trace

graph
graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.762047230368452 > 12.72044796463289.
IsChangePoint: Marked as a change because one of 4/9/2023 4:09:45 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.12449951243706 (T) = (0 -18.012537506242346) / Math.Sqrt((3.0823232902543185 / (31)) + (0.05254668041948599 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.31520201852300755 = (13.695643142694312 - 18.012537506242346) / 13.695643142694312 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.865670862503706 > 2.47964434493585.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -174.8051686724907 (T) = (0 -14.849889741368294) / Math.Sqrt((0.05964328802689774 / (31)) + (0.02002248505391683 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -5.8409967413071335 = (2.1707201893113126 - 14.849889741368294) / 2.1707201893113126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterDecimal(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 85.75961971334873 > 52.82228194256532.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -166.71470021044755 (T) = (0 -85.23693451806466) / Math.Sqrt((0.598314175101614 / (31)) + (0.15276926941382157 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.7059310913792499 = (49.96505131350315 - 85.23693451806466) / 49.96505131350315 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.032124329722276 > 1.6895710431085587.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -64.23582465898374 (T) = (0 -10.115425400080673) / Math.Sqrt((0.2759353496132015 / (31)) + (0.04072984011584353 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -3.888083649920233 = (2.0694051339006325 - 10.115425400080673) / 2.0694051339006325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 0)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.587405248824826 > 1.6787238541271141.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -146.79517772147875 (T) = (0 -9.582075910972657) / Math.Sqrt((0.002608316071381241 / (31)) + (0.017366825278672646 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -5.101337743963003 = (1.5704877050043142 - 9.582075910972657) / 1.5704877050043142 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_Dictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Clone - Duration of single invocation 16.64 μs 26.65 μs 1.60 0.01 True 132365.46651180796 229916.81521093287 1.73698488941164) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_Dictionary*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_Dictionary.Clone(Items: 3000)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 26.654914892093203 > 17.443096664666566.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -33.78586050576494 (T) = (0 -26341.90298849711) / Math.Sqrt((15385.563432871779 / (30)) + (501189.85432585713 / (6))) is less than -2.0322445093148245 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (6) - 2, .025) and -0.5919215144858052 = (16547.237253091345 - 26341.90298849711) / 16547.237253091345 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollection<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FrozenDictionary - Duration of single invocation 2.98 μs 4.61 μs 1.54 0.01 True Trace Trace
ImmutableStack - Duration of single invocation 3.86 μs 4.17 μs 1.08 0.02 False Trace Trace
Dictionary - Duration of single invocation 2.91 μs 4.51 μs 1.55 0.01 True Trace Trace

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<Int32>.FrozenDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.609846951160307 > 3.122794848267486.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -132.72671382153962 (T) = (0 -4570.134690513872) / Math.Sqrt((134.40010136323713 / (31)) + (848.2963282718607 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.5398338555959878 = (2967.9401280244065 - 4570.134690513872) / 2967.9401280244065 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.CtorFromCollection<Int32>.ImmutableStack(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.173858382144125 > 4.059562289126971.
IsChangePoint: Marked as a change because one of 1/30/2023 9:42:46 PM, 2/9/2023 4:27:10 AM, 2/20/2023 10:31:48 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -19.980554920087325 (T) = (0 -4219.2249538380165) / Math.Sqrt((328.4257591702791 / (31)) + (1772.2695157733767 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.09031687210232284 = (3869.7236205311656 - 4219.2249538380165) / 3869.7236205311656 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.CtorFromCollection<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.514597771681586 > 3.0771396600023744.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -153.3583992850678 (T) = (0 -4525.412222101016) / Math.Sqrt((111.91706382949505 / (31)) + (623.783836219791 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.5419780088113653 = (2934.809832722214 - 4525.412222101016) / 2934.809832722214 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Decimal

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 75.92 ns 83.34 ns 1.10 0.02 False 771.1966658733542 812.2590628520107 1.0532450395544108) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Decimal*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Decimal.ToString(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 83.34137516326994 > 79.6403110107157.
IsChangePoint: Marked as a change because one of 2/17/2023 5:12:50 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -13.612634544106168 (T) = (0 -82.96994973109237) / Math.Sqrt((1.419027731816655 / (31)) + (0.8281283224183011 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.0756597431833429 = (77.1340103196094 - 82.96994973109237) / 77.1340103196094 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ctor - Duration of single invocation 341.88 μs 366.93 μs 1.07 0.00 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Ctor(Pattern: "(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])", Options: NonBacktracking)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 366.92863436123343 > 361.7632134880091.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -26.2978592393311 (T) = (0 -366759.71848137816) / Math.Sqrt((7041973.221191616 / (31)) + (2840242.092644557 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.06384576228074225 = (344748.95843462745 - 366759.71848137816) / 344748.95843462745 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.WriteJson<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SerializeToWriter - Duration of single invocation 57.48 ns 64.91 ns 1.13 0.28 False 528.6013947809427 572.5430915367768 1.0831282270339904) Trace Trace
SerializeToWriter - Duration of single invocation 56.67 ns 61.83 ns 1.09 0.28 False 529.1603192433358 572.5480247783256 1.0819934979195556) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.WriteJson&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToWriter(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 64.90666431613528 > 59.55483540091958.
IsChangePoint: Marked as a change because one of 3/20/2023 10:25:17 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -4.342866457279257 (T) = (0 -63.87750469320892) / Math.Sqrt((9.04623316955244 / (31)) + (9.050151560890365 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.1003751929355774 = (58.050658632894766 - 63.87750469320892) / 58.050658632894766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Text.Json.Serialization.Tests.WriteJson<Int32>.SerializeToWriter(Mode: SourceGen)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 61.832305171814006 > 60.79854199182073.
IsChangePoint: Marked as a change because one of 3/20/2023 11:25:43 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -6.362985327256353 (T) = (0 -63.91543020174091) / Math.Sqrt((4.296972788389164 / (31)) + (4.882179251024649 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.10760404136752168 = (57.706028340982456 - 63.91543020174091) / 57.706028340982456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Append_Primitives - Duration of single invocation 1.61 μs 1.73 μs 1.07 0.00 False 12337.263899263686 12900.89232377539 1.0456850424140918) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.Append_Primitives

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.7326337352271892 > 1.6956151780623874.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.67596168390295 (T) = (0 -1718.8055243365175) / Math.Sqrt((152.22542060524162 / (30)) + (100.02068232886876 / (5))) is less than -2.034515297446192 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (30) + (5) - 2, .025) and -0.057544536244142186 = (1625.2795654742224 - 1718.8055243365175) / 1625.2795654742224 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollection<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FrozenDictionary - Duration of single invocation 6.16 μs 7.97 μs 1.29 0.01 True 38934.19768576607 54442.663817663815 1.3983250472262196) Trace Trace
Dictionary - Duration of single invocation 6.10 μs 7.75 μs 1.27 0.00 True 37888.0380794702 53583.419421487604 1.4142569036986377) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollection&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollection<String>.FrozenDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.965227523324852 > 6.481048705622619.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -64.1229773246127 (T) = (0 -7961.620790386102) / Math.Sqrt((6772.646668516648 / (31)) + (3126.9978262840664 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.28047415134744613 = (6217.713010456376 - 7961.620790386102) / 6217.713010456376 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.CtorFromCollection<String>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.750633482290632 > 6.396525421868904.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -73.97378905243747 (T) = (0 -7769.770320688808) / Math.Sqrt((12595.313861739534 / (31)) + (320.22242048812205 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.256476468560336 = (6183.777026553764 - 7769.770320688808) / 6183.777026553764 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryAddDefaultSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 13.38 μs 19.35 μs 1.45 0.20 False 80231.35867963455 122361.11111111112 1.5251033152722933) Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddDefaultSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddDefaultSize<Int32>.Dictionary(Count: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.35125 > 14.105542235540574.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -59.57449616593256 (T) = (0 -19239.848768947635) / Math.Sqrt((211865.87145343257 / (31)) + (11933.945027721453 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.41014821391030876 = (13643.848624674689 - 19239.848768947635) / 13643.848624674689 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 5.86 μs 7.34 μs 1.25 0.10 False Trace Trace
IDictionary - Duration of single invocation 7.36 μs 8.24 μs 1.12 0.21 True 36355.75777272235 52811.38623147627 1.4526278495314584) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;Int32, Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.341722557896865 > 6.121403036167097.
IsChangePoint: Marked as a change because one of 2/24/2023 12:59:32 PM, 3/15/2023 12:03:45 AM, 3/31/2023 8:05:26 PM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -7.069912677444894 (T) = (0 -6931.38625989033) / Math.Sqrt((60854.709171491566 / (31)) + (236064.77489385288 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.2615169441726428 = (5494.485263879061 - 6931.38625989033) / 5494.485263879061 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.ContainsKeyTrue<Int32, Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 8.24040731200984 > 6.42953850455.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -18.459448492605762 (T) = (0 -8559.99269265322) / Math.Sqrt((257046.26299382639 / (31)) + (138263.24588812608 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.6174362552525715 = (5292.321514900463 - 8559.99269265322) / 5292.321514900463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 13.09 μs 20.32 μs 1.55 0.21 False Trace Trace
IDictionary - Duration of single invocation 13.77 μs 20.59 μs 1.50 0.04 True 83806.37023021128 125332.65674334687 1.4955027451859002) Trace Trace

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndClear<Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.317901534803283 > 14.878679541747951.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -26.429405147779264 (T) = (0 -19445.284574115012) / Math.Sqrt((515148.3007195261 / (31)) + (191752.73351456842 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.4276772196336637 = (13620.224730562413 - 19445.284574115012) / 13620.224730562413 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

System.Collections.CreateAddAndClear<Int32>.IDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.58564808707124 > 14.39890747346222.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -65.79911842040511 (T) = (0 -20213.65986262697) / Math.Sqrt((32704.472491773617 / (31)) + (51015.013808779884 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.46677949613249686 = (13780.98065586883 - 20213.65986262697) / 13780.98065586883 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryAddGiventSize<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Dictionary - Duration of single invocation 8.01 μs 11.62 μs 1.45 0.12 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryAddGiventSize&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryAddGiventSize<Int32>.Dictionary(Count: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 11.618850606736007 > 8.425977923227295.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/16/2023 6:27:39 AM, 3/25/2023 12:18:49 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -54.87670780966784 (T) = (0 -11668.920884022207) / Math.Sqrt((58684.1903053703 / (31)) + (12601.556157115567 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.422844436635385 = (8201.122050711198 - 11668.920884022207) / 8201.122050711198 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Apr 18, 2023 •
edited
Loading

Run Information

Name Value
Architecture x86
OS Windows 10.0.18362
Queue TigerWindows
Baseline 7526a4c6a0a095670f1135d983a349b4e867f2e2
Compare c1b7a9feb6f3b4d9ca27dc4f74d8260e4edb73e8
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.AddGivenSize<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
HashSet - Duration of single invocation 15.29 μs 16.91 μs 1.11 0.02 False Trace Trace

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.AddGivenSize&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.AddGivenSize<String>.HashSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.907570846617073 > 16.06531610084773.
IsChangePoint: Marked as a change because one of 2/9/2023 4:27:10 AM, 2/16/2023 1:21:52 PM, 3/15/2023 12:03:45 AM, 4/12/2023 8:39:35 PM, 4/18/2023 4:36:42 PM falls between 4/9/2023 11:04:55 PM and 4/18/2023 4:36:42 PM.
IsRegressionStdDev: Marked as regression because -16.274413517400827 (T) = (0 -16701.911939758163) / Math.Sqrt((38308.38432351742 / (31)) + (33562.81162912648 / (6))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (31) + (6) - 2, .025) and -0.08757817368348367 = (15356.976026091985 - 16701.911939758163) / 15356.976026091985 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Baseline JIT Disasm

Complete output

Compare JIT Disasm

Complete output

JIT Disasm Diff

Link

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@DrewScoggins DrewScoggins mentioned this issue Sep 5, 2023
Closed
@DrewScoggins DrewScoggins mentioned this issue Sep 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@EgorBo EgorBo

Labels
arch-x86 branch-refs/heads/main kind-micro os-windows perf-regression runtime-coreclr untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@EgorBo