[Perf] Windows/arm64: 8 Regressions on 4/18/2023 2:18:01 AM

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.19041
Queue	SurfaceWindows
Baseline	e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare	90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int16

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Parse - Duration of single invocation	14.34 ns	16.75 ns	1.17	0.09	False
TryParse - Duration of single invocation	15.54 ns	17.28 ns	1.11	0.10	False
TryParse - Duration of single invocation	14.75 ns	16.13 ns	1.09	0.10	False
Parse - Duration of single invocation	15.41 ns	17.41 ns	1.13	0.10	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.Parse(value: "32767")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.754057448366503 > 15.304379942656118.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -18.447751132126594 (T) = (0 -16.542891763708074) / Math.Sqrt((0.08381383234826988 / (17)) + (0.03885769935260771 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.12400357441415305 = (14.71782843068846 - 16.542891763708074) / 14.71782843068846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.TryParse(value: "-32768")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.28025893244643 > 16.313391482483738.
IsChangePoint: Marked as a change because one of 4/17/2023 2:42:45 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -17.439763396159954 (T) = (0 -17.62135380046994) / Math.Sqrt((0.0926058495951353 / (16)) + (0.0615648089790755 / (8))) is less than -2.073873067894501 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (16) + (8) - 2, .025) and -0.12984356054482402 = (15.59627758729068 - 17.62135380046994) / 15.59627758729068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.TryParse(value: "32767")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.12935999454518 > 15.47039981662087.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -11.214659857645975 (T) = (0 -16.35435326865967) / Math.Sqrt((0.23876303257302917 / (17)) + (0.04562361049108938 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.10663964948746439 = (14.778390848578505 - 16.35435326865967) / 14.778390848578505 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_Int16.Parse(value: "-32768")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.409051613705525 > 16.254917857342853.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -17.113625381644614 (T) = (0 -17.83845005692101) / Math.Sqrt((0.22694551836929977 / (17)) + (0.03608347969523643 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.14706749213669204 = (15.551351754980484 - 17.83845005692101) / 15.551351754980484 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.19041
Queue	SurfaceWindows
Baseline	e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare	90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt16

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryParse - Duration of single invocation	14.77 ns	16.19 ns	1.10	0.13	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt16.TryParse(value: "65535")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.187123245473803 > 15.634913978147972.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -11.3979815049189 (T) = (0 -16.269752000066212) / Math.Sqrt((0.1974651127078325 / (17)) + (0.008947509095466905 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.085841316939061 = (14.9835447834403 - 16.269752000066212) / 14.9835447834403 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.19041
Queue	SurfaceWindows
Baseline	e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare	90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_SByte

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Parse - Duration of single invocation	12.68 ns	13.99 ns	1.10	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.Parse(value: "-128")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.990505926447984 > 13.282965365495125.
IsChangePoint: Marked as a change because one of 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -8.47343527001088 (T) = (0 -13.677202572339432) / Math.Sqrt((0.125513670073459 / (17)) + (0.06343826774430601 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.0830293754513892 = (12.628653370218139 - 13.677202572339432) / 12.628653370218139 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.19041
Queue	SurfaceWindows
Baseline	e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare	90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<Int32, Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Dictionary - Duration of single invocation	5.07 μs	6.06 μs	1.19	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue<Int32, Int32>*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<Int32, Int32>.Dictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 6.063808507273878 > 5.319455539772728.
IsChangePoint: Marked as a change because one of 1/5/2023 2:25:12 AM, 1/15/2023 4:47:52 PM, 3/25/2023 12:18:49 AM, 4/9/2023 11:04:55 PM, 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -7.153923736205714 (T) = (0 -5989.199610614853) / Math.Sqrt((170017.44213104408 / (17)) + (2892.456320872582 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.13842479049524167 = (5260.953258062317 - 5989.199610614853) / 5260.953258062317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

---

Run Information

Name	Value
Architecture	arm64
OS	Windows 10.0.19041
Queue	SurfaceWindows
Baseline	e3f55cbfd83162bc85d73d69c765e7f44c478e6d
Compare	90d0d5ce4e4fdf17209d6a4e4a96d62865e2b88f
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrueComparer<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ImmutableSortedSet - Duration of single invocation	312.35 μs	341.66 μs	1.09	0.04	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer<String>*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrueComparer<String>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 341.6607725155279 > 330.8255222596154.
IsChangePoint: Marked as a change because one of 1/21/2023 1:51:00 AM, 1/26/2023 1:49:07 PM, 2/9/2023 1:41:04 PM, 3/5/2023 11:54:18 PM, 4/17/2023 8:43:19 PM, 4/20/2023 3:04:43 PM falls between 4/11/2023 11:54:05 PM and 4/20/2023 3:04:43 PM.
IsRegressionStdDev: Marked as regression because -13.249226506038088 (T) = (0 -343910.5188030376) / Math.Sqrt((19798152.08201066 / (17)) + (20012243.50638626 / (8))) is less than -2.0686576104106944 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (17) + (8) - 2, .025) and -0.07963932093982869 = (318542.0465268555 - 343910.5188030376) / 318542.0465268555 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kunalspathak]

dotnet/runtime#84582