Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 74 Regressions on 1/15/2023 10:25:39 PM #12080

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 4 comments
Closed

[Perf] Linux/arm64: 74 Regressions on 1/15/2023 10:25:39 PM #12080

performanceautofiler bot opened this issue Jan 24, 2023 · 4 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_Int64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 651.01 ns 859.40 ns 1.32 0.10 False
ToString - Duration of single invocation 482.65 ns 713.44 ns 1.48 0.18 False
ToString - Duration of single invocation 254.81 ns 457.91 ns 1.80 0.05 False
TryFormat - Duration of single invocation 690.54 ns 920.76 ns 1.33 0.06 False
TryFormat - Duration of single invocation 278.69 ns 492.14 ns 1.77 0.08 False
TryFormat - Duration of single invocation 515.52 ns 722.77 ns 1.40 0.19 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int64.ToString(value: -9223372036854775808)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -33.095376716597784 (T) = (0 -865.2700159776588) / Math.Sqrt((26.7361599658936 / (3)) + (237.9035898515841 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3342514500045192 = (648.5059588840829 - 865.2700159776588) / 648.5059588840829 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ToString(value: 9223372036854775807)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -30.066255784866158 (T) = (0 -722.7462823488792) / Math.Sqrt((51.79595366498003 / (3)) + (304.2707066250194 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4797134624241048 = (488.43664716333495 - 722.7462823488792) / 488.43664716333495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.ToString(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -195.09654879432358 (T) = (0 -456.41526839401934) / Math.Sqrt((1.5554958294189751 / (3)) + (3.7873584276129653 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7857060016433914 = (255.59373602036325 - 456.41526839401934) / 255.59373602036325 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryFormat(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -36.3267430311068 (T) = (0 -914.4111772346594) / Math.Sqrt((51.209639554803545 / (3)) + (128.35536810415692 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3095657366782918 = (698.2552701432611 - 914.4111772346594) / 698.2552701432611 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -71.74210042293288 (T) = (0 -484.5741103214599) / Math.Sqrt((5.512854784108389 / (3)) + (44.16186911567508 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7318104690814088 = (279.8078190268066 - 484.5741103214599) / 279.8078190268066 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int64.TryFormat(value: 9223372036854775807)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -25.588873538238158 (T) = (0 -730.7510879648489) / Math.Sqrt((152.53667936500136 / (3)) + (85.95428697386826 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.38545530966248626 = (527.444720063088 - 730.7510879648489) / 527.444720063088 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_Enum

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InterpolateIntoString - Duration of single invocation 10.94 μs 12.27 μs 1.12 0.07 False
ToString_NonFlags_Small - Duration of single invocation 1.06 μs 1.36 μs 1.28 0.07 False
ToString_Flags - Duration of single invocation 1.16 μs 1.33 μs 1.15 0.14 False
ToString_Format_Flags_Large - Duration of single invocation 902.54 ns 1.18 μs 1.30 0.21 False
StringFormat - Duration of single invocation 8.58 μs 10.11 μs 1.18 0.07 False
ToString_Flags - Duration of single invocation 1.20 μs 1.43 μs 1.19 0.07 False
InterpolateIntoSpan_NonFlags - Duration of single invocation 10.73 μs 12.10 μs 1.13 0.14 False
InterpolateIntoSpan_Flags - Duration of single invocation 10.27 μs 11.95 μs 1.16 0.07 False
InterpolateIntoStringBuilder_NonFlags - Duration of single invocation 10.97 μs 11.77 μs 1.07 0.21 False
ToString_NonFlags_Large - Duration of single invocation 928.58 ns 1.20 μs 1.30 0.14 False
InterpolateIntoStringBuilder_Flags - Duration of single invocation 10.78 μs 11.60 μs 1.08 0.07 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.InterpolateIntoString(value: 32)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -17.177807615232204 (T) = (0 -12337.08621807441) / Math.Sqrt((6185.100276183106 / (3)) + (28266.01644288279 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12201339175610751 = (10995.489277329523 - 12337.08621807441) / 10995.489277329523 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_NonFlags_Small(value: -1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -27.541687571200402 (T) = (0 -1327.0352187605993) / Math.Sqrt((81.79439712410466 / (3)) + (423.3172028492839 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.24131592739767585 = (1069.0551772284332 - 1327.0352187605993) / 1069.0551772284332 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_Flags(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -15.43029237067653 (T) = (0 -1371.4184426303113) / Math.Sqrt((273.7942960707263 / (3)) + (899.0560605404988 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.20015025601432546 = (1142.7056201985608 - 1371.4184426303113) / 1142.7056201985608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -13.7999906797011 (T) = (0 -1171.529776057601) / Math.Sqrt((710.6974121535924 / (3)) + (507.1937143459526 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2613252616109037 = (928.8086203564811 - 1171.529776057601) / 928.8086203564811 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.StringFormat(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.2260905759331 (T) = (0 -9720.872222255932) / Math.Sqrt((6972.31771952942 / (3)) + (42234.36819946623 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12991814547534639 = (8603.164982509816 - 9720.872222255932) / 8603.164982509816 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_Flags(value: 36)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.60590109145653 (T) = (0 -1452.9357068415047) / Math.Sqrt((860.9770571999692 / (3)) + (967.4027692841173 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2178846158604931 = (1192.9994745971374 - 1452.9357068415047) / 1192.9994745971374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_NonFlags(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -7.569973995795917 (T) = (0 -11885.385673088953) / Math.Sqrt((11825.344733907059 / (3)) + (112769.48968650837 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.09913019004768546 = (10813.446651459286 - 11885.385673088953) / 10813.446651459286 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoSpan_Flags(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -7.619315676823492 (T) = (0 -11789.987124046515) / Math.Sqrt((76554.24917959468 / (3)) + (50643.905352137845 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.132448192493553 = (10411.060922871873 - 11789.987124046515) / 10411.060922871873 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_NonFlags(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -12.06516781542372 (T) = (0 -12035.942375315399) / Math.Sqrt((5142.016405430455 / (3)) + (51493.64012834289 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.1055453792561089 = (10886.882258432533 - 12035.942375315399) / 10886.882258432533 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.ToString_NonFlags_Large(value: 42)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -26.430989632963186 (T) = (0 -1187.904119672497) / Math.Sqrt((177.64188723732553 / (3)) + (264.38984780682273 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.28060643720911754 = (927.6106110018853 - 1187.904119672497) / 927.6106110018853 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Enum.InterpolateIntoStringBuilder_Flags(value: 32)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -11.430229941794801 (T) = (0 -11636.042278074277) / Math.Sqrt((4119.94214008344 / (3)) + (25544.61884455195 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.07482548929374494 = (10825.982816727004 - 11636.042278074277) / 10825.982816727004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Deep

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteDeepUtf16 - Duration of single invocation 151.67 ms 173.07 ms 1.14 0.01 False
WriteDeepUtf8 - Duration of single invocation 205.42 ms 222.09 ms 1.08 0.01 False
WriteDeepUtf16 - Duration of single invocation 196.96 ms 218.14 ms 1.11 0.03 False
WriteDeepUtf8 - Duration of single invocation 199.31 ms 219.45 ms 1.10 0.03 False
WriteDeepUtf8 - Duration of single invocation 151.93 ms 170.97 ms 1.13 0.03 False
WriteDeepUtf16 - Duration of single invocation 152.34 ms 162.52 ms 1.07 0.08 False
WriteDeepUtf16 - Duration of single invocation 201.36 ms 223.42 ms 1.11 0.01 False
WriteDeepUtf8 - Duration of single invocation 146.65 ms 165.21 ms 1.13 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Deep*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: False)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -20.95542695073325 (T) = (0 -171275693.1252747) / Math.Sqrt((1635224229119.0781 / (3)) + (2492561183545.021 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.13140729714259708 = (151382878.25952384 - 171275693.1252747) / 151382878.25952384 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: True, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -16.73831426654424 (T) = (0 -224479007.76713768) / Math.Sqrt((2894850251741.879 / (3)) + (3400601215370.486 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.09867360685482479 = (204318194.56349206 - 224479007.76713768) / 204318194.56349206 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: True, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -20.40066484030584 (T) = (0 -217825328.14201993) / Math.Sqrt((2097286694964.6772 / (3)) + (2908033348946.6255 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.1097225864985254 = (196288091.08888888 - 217825328.14201993) / 196288091.08888888 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: True, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -17.749747299324902 (T) = (0 -217998984.19183677) / Math.Sqrt((2924868027481.7627 / (3)) + (2028955682295.9502 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10079941052200278 = (198036973.95555556 - 217998984.19183677) / 198036973.95555556 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -18.123757176587247 (T) = (0 -170707883.54149657) / Math.Sqrt((1717438624248.9868 / (3)) + (4166032651884.2524 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12958867949807887 = (151123932.66666666 - 170707883.54149657) / 151123932.66666666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -6.402736807338654 (T) = (0 -164076576.29489797) / Math.Sqrt((20305267561746.72 / (3)) + (1486350044866.8816 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.11495495196796798 = (147159825.6102564 - 164076576.29489797) / 147159825.6102564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf16(Formatted: True, SkipValidation: False)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -19.95623596472667 (T) = (0 -224217865.9802721) / Math.Sqrt((1397914486633.5112 / (3)) + (5390138137686.538 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10981641284739044 = (202031492.2222222 - 224217865.9802721) / 202031492.2222222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Deep.WriteDeepUtf8(Formatted: False, SkipValidation: True)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -16.69303026034224 (T) = (0 -165218903.11206174) / Math.Sqrt((3596417388831.634 / (3)) + (1336948890361.8418 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.13521663047183105 = (145539537.28055558 - 165218903.11206174) / 145539537.28055558 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_UInt32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat - Duration of single invocation 176.43 ns 370.08 ns 2.10 0.13 False
TryFormat - Duration of single invocation 262.03 ns 467.97 ns 1.79 0.12 False
TryFormat - Duration of single invocation 347.43 ns 538.99 ns 1.55 0.14 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.TryFormat(value: 0)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -72.50949226364443 (T) = (0 -365.66966350232985) / Math.Sqrt((17.516547211463834 / (3)) + (6.70049755960034 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -1.0701081181635212 = (176.64278512502554 - 365.66966350232985) / 176.64278512502554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -60.02612042668496 (T) = (0 -459.2273344134799) / Math.Sqrt((22.244564983814016 / (3)) + (22.751626998395565 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7448022117979518 = (263.19735916672397 - 459.2273344134799) / 263.19735916672397 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt32.TryFormat(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -52.89791700144702 (T) = (0 -543.6890265527547) / Math.Sqrt((31.255793282670698 / (3)) + (28.37266393979334 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5876195786450963 = (342.45548106477054 - 543.6890265527547) / 342.45548106477054 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_UInt64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 243.85 ns 444.44 ns 1.82 0.05 False
TryFormat - Duration of single invocation 531.82 ns 730.87 ns 1.37 0.14 False
TryFormat - Duration of single invocation 194.01 ns 392.93 ns 2.03 0.22 False
ToString - Duration of single invocation 514.92 ns 736.82 ns 1.43 0.10 False
TryFormat - Duration of single invocation 278.91 ns 488.58 ns 1.75 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt64*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt64.ToString(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -105.3322833916439 (T) = (0 -443.49382043185733) / Math.Sqrt((2.174137166241629 / (3)) + (20.132711688011877 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.8204531286473284 = (243.61726948794967 - 443.49382043185733) / 243.61726948794967 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.TryFormat(value: 18446744073709551615)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -30.718440701845573 (T) = (0 -740.6249745803212) / Math.Sqrt((136.64216161111813 / (3)) + (46.890130822746436 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4281572067265047 = (518.5878494972666 - 740.6249745803212) / 518.5878494972666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.TryFormat(value: 0)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -38.8697257633278 (T) = (0 -393.2510046008057) / Math.Sqrt((43.788002856636034 / (3)) + (83.25272555084639 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -1.035487622027746 = (193.19744337676215 - 393.2510046008057) / 193.19744337676215 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.ToString(value: 18446744073709551615)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -25.545731732715854 (T) = (0 -723.9060356751171) / Math.Sqrt((97.53412725649694 / (3)) + (253.6386295303145 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4136056341154959 = (512.0990028651596 - 723.9060356751171) / 512.0990028651596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_UInt64.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -70.87280419724601 (T) = (0 -483.66202660199446) / Math.Sqrt((7.510880125983298 / (3)) + (40.86742584564666 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7337695181224175 = (278.96558426391954 - 483.66202660199446) / 278.96558426391954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_Int16

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 366.19 ns 583.23 ns 1.59 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int16*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int16.ToString(value: -32768)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -69.97551902301667 (T) = (0 -578.0321373755036) / Math.Sqrt((1.2112074709057938 / (3)) + (60.56649017029201 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5730802141323703 = (367.4524237114737 - 578.0321373755036) / 367.4524237114737 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Buffers.Text.Tests.Utf8FormatterTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FormatterUInt64 - Duration of single invocation 308.27 ns 507.49 ns 1.65 0.14 False
FormatterUInt32 - Duration of single invocation 387.15 ns 597.90 ns 1.54 0.18 False
FormatterUInt64 - Duration of single invocation 536.10 ns 758.49 ns 1.41 0.26 False
FormatterInt32 - Duration of single invocation 401.24 ns 612.39 ns 1.53 0.17 False
FormatterInt64 - Duration of single invocation 555.31 ns 731.40 ns 1.32 0.21 False
FormatterInt32 - Duration of single invocation 396.76 ns 615.37 ns 1.55 0.10 False
FormatterInt64 - Duration of single invocation 550.13 ns 807.81 ns 1.47 0.29 False
FormatterInt32 - Duration of single invocation 319.18 ns 538.63 ns 1.69 0.06 False
FormatterInt64 - Duration of single invocation 315.97 ns 529.25 ns 1.68 0.13 False
FormatterUInt32 - Duration of single invocation 306.22 ns 503.56 ns 1.64 0.06 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8FormatterTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 12345)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -65.23699608649991 (T) = (0 -509.1422551434549) / Math.Sqrt((18.691522447014876 / (3)) + (25.935738165638075 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6775140654576423 = (303.509976832627 - 509.1422551434549) / 303.509976832627 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 4294967295)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -50.099743831126816 (T) = (0 -595.1469771357336) / Math.Sqrt((11.061611160216005 / (3)) + (96.44534513947775 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.542735850278376 = (385.77373892513316 - 595.1469771357336) / 385.77373892513316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt64(value: 18446744073709551615)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -39.11703218774737 (T) = (0 -748.2456546783447) / Math.Sqrt((4.629383871717714 / (3)) + (190.78383306260943 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.389941037669949 = (538.329061736805 - 748.2456546783447) / 538.329061736805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -22.17943836076709 (T) = (0 -613.5552722024913) / Math.Sqrt((221.0064700268349 / (3)) + (27.370667492459432 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4671292922650424 = (418.2012283697559 - 613.5552722024913) / 418.2012283697559 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 9223372036854775807)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -37.831122772643695 (T) = (0 -747.9837124442156) / Math.Sqrt((4.247229898124068 / (3)) + (175.92312503769844 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3524429485776402 = (553.0611943600783 - 747.9837124442156) / 553.0611943600783 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -73.43949711440894 (T) = (0 -612.1415538604446) / Math.Sqrt((8.551390196490708 / (3)) + (38.46271896093865 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5303925680586029 = (399.9898892850635 - 612.1415538604446) / 399.9898892850635 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: -9223372036854775808)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -14.777160843416494 (T) = (0 -760.6869260305388) / Math.Sqrt((360.9557040590253 / (3)) + (477.5440775804548 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.36377008453434134 = (557.7823818376797 - 760.6869260305388) / 557.7823818376797 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -52.408866236334276 (T) = (0 -528.5340581582793) / Math.Sqrt((30.26948283058484 / (3)) + (38.47694751003209 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6433240297165501 = (321.62497998002493 - 528.5340581582793) / 321.62497998002493 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterInt64(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -48.1061421874436 (T) = (0 -525.1377648896603) / Math.Sqrt((28.95316979094856 / (3)) + (57.736734334282694 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6328241337872349 = (321.613181740299 - 525.1377648896603) / 321.613181740299 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Buffers.Text.Tests.Utf8FormatterTests.FormatterUInt32(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -52.70650287812027 (T) = (0 -510.72190879539363) / Math.Sqrt((36.023294655797535 / (3)) + (27.668923146527046 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7015187014651699 = (300.1565062761951 - 510.72190879539363) / 300.1565062761951 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_ToStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 3.88 ms 4.27 ms 1.10 0.07 False
SystemTextJson_SourceGen_ - Duration of single invocation 4.05 ms 4.41 ms 1.09 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_ToStream&lt;CollectionsOfPrimitives&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_ToStream<CollectionsOfPrimitives>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -21.18911028968416 (T) = (0 -4258635.132225181) / Math.Sqrt((837612563.291029 / (3)) + (545149717.5258384 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10377882879440567 = (3858232.3026404157 - 4258635.132225181) / 3858232.3026404157 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_ToStream&lt;CollectionsOfPrimitives&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -19.442134498403274 (T) = (0 -4438660.114759882) / Math.Sqrt((957855001.0682694 / (3)) + (880545963.5468512 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.10181672563142076 = (4028492.2269774117 - 4438660.114759882) / 4028492.2269774117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_SByte

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 329.94 ns 534.02 ns 1.62 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_SByte*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_SByte.ToString(value: -128)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -51.52633281895563 (T) = (0 -535.2149956250745) / Math.Sqrt((2.6165759489400577 / (3)) + (106.90498047297075 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6308474241825678 = (328.1821387389082 - 535.2149956250745) / 328.1821387389082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Xml.Tests.Perf_XmlConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TimeSpan_ToString - Duration of single invocation 2.27 μs 3.06 μs 1.35 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Tests.Perf_XmlConvert*'

Payloads

Baseline
Compare

Histogram

System.Xml.Tests.Perf_XmlConvert.TimeSpan_ToString

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -47.80029049380426 (T) = (0 -3079.272711458628) / Math.Sqrt((253.939381670211 / (3)) + (1499.919188772674 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.36684152671998427 = (2252.8381317532635 - 3079.272711458628) / 2252.8381317532635 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Jan 24, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_Int32

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ToString - Duration of single invocation 471.74 ns 686.53 ns 1.46 0.09 False
TryFormat - Duration of single invocation 178.51 ns 381.81 ns 2.14 0.09 False
TryFormat - Duration of single invocation 271.61 ns 474.34 ns 1.75 0.07 False
TryFormat - Duration of single invocation 530.02 ns 753.64 ns 1.42 0.07 False
TryFormat - Duration of single invocation 342.76 ns 556.47 ns 1.62 0.21 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: -2147483648)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/8/2022 2:00:38 AM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -83.74623844374649 (T) = (0 -685.6706868972303) / Math.Sqrt((0.4597927440166369 / (3)) + (44.49864535796977 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.4527193627172953 = (471.99115293313844 - 685.6706868972303) / 471.99115293313844 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: 4)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -43.199208485963254 (T) = (0 -374.4578307021041) / Math.Sqrt((7.062393701429039 / (3)) + (128.67509556856666 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -1.106759931284132 = (177.74110146183625 - 374.4578307021041) / 177.74110146183625 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: 12345)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -73.46459272146235 (T) = (0 -467.21286875388415) / Math.Sqrt((6.479117261381439 / (3)) + (35.923510967841146 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7379089731878166 = (268.836214072181 - 467.21286875388415) / 268.836214072181 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: -2147483648)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -34.83344777714515 (T) = (0 -737.963364981271) / Math.Sqrt((53.89425730529699 / (3)) + (123.27578536239851 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3918628482918174 = (530.1983351929729 - 737.963364981271) / 530.1983351929729 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Int32.TryFormat(value: 2147483647)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -34.19534606799376 (T) = (0 -554.3503933213258) / Math.Sqrt((104.18154331298386 / (3)) + (18.789770179894557 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6059053019367243 = (345.1949455878739 - 554.3503933213258) / 345.1949455878739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Basic

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
WriteBasicUtf16 - Duration of single invocation 27.24 μs 29.12 μs 1.07 0.03 False
WriteBasicUtf8 - Duration of single invocation 55.70 ms 76.09 ms 1.37 0.03 False
WriteBasicUtf16 - Duration of single invocation 70.22 ms 91.04 ms 1.30 0.05 False
WriteBasicUtf16 - Duration of single invocation 23.14 μs 24.84 μs 1.07 0.07 False
WriteBasicUtf16 - Duration of single invocation 67.94 ms 88.09 ms 1.30 0.03 False
WriteBasicUtf16 - Duration of single invocation 24.01 μs 26.93 μs 1.12 0.03 False
WriteBasicUtf8 - Duration of single invocation 52.94 ms 73.36 ms 1.39 0.03 False
WriteBasicUtf8 - Duration of single invocation 68.34 ms 87.80 ms 1.28 0.03 False
WriteBasicUtf8 - Duration of single invocation 70.71 ms 90.37 ms 1.28 0.03 False
WriteBasicUtf16 - Duration of single invocation 55.27 ms 75.36 ms 1.36 0.03 False
WriteBasicUtf16 - Duration of single invocation 28.09 μs 30.08 μs 1.07 0.04 False
WriteBasicUtf16 - Duration of single invocation 52.71 ms 73.11 ms 1.39 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Basic*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 10)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -11.9041549068412 (T) = (0 -29348.485604327354) / Math.Sqrt((78414.84188823469 / (3)) + (87077.16655727038 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.08656400528444384 = (27010.36060608728 - 29348.485604327354) / 27010.36060608728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -55.896880357461164 (T) = (0 -75553055.01621532) / Math.Sqrt((256929473651.9722 / (3)) + (332818241248.5633 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.36986924042795727 = (55153479.46101191 - 75553055.01621532) / 55153479.46101191 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -86.60497137727823 (T) = (0 -91246581.35247253) / Math.Sqrt((11338629419.816021 / (3)) + (384429202134.1884 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.29862095588269233 = (70264214.46468253 - 91246581.35247253) / 70264214.46468253 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -8.840688054091874 (T) = (0 -25079.910430904678) / Math.Sqrt((105112.20627377977 / (3)) + (194118.16025182212 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.09686927925475396 = (22864.9948587718 - 25079.910430904678) / 22864.9948587718 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -63.90706080730794 (T) = (0 -88623710.52495421) / Math.Sqrt((260841463557.06583 / (3)) + (162241549446.4839 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.3145769901625613 = (67416143.13057081 - 88623710.52495421) / 67416143.13057081 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -11.912204224805336 (T) = (0 -26207.503822869206) / Math.Sqrt((31101.18148186426 / (3)) + (135831.0865367945 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.08510149789784927 = (24152.12205829649 - 26207.503822869206) / 24152.12205829649 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: False, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -40.85567433228436 (T) = (0 -73309721.60743067) / Math.Sqrt((375840138919.8826 / (3)) + (925324730148.3727 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.39427845981034954 = (52578967.344444446 - 73309721.60743067) / 52578967.344444446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -79.1960283678828 (T) = (0 -88312747.76176085) / Math.Sqrt((78064403854.25465 / (3)) + (269174268977.9476 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2948445410960515 = (68203359.52222222 - 88312747.76176085) / 68203359.52222222 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf8(Formatted: True, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -73.77840393839995 (T) = (0 -90929163.16629383) / Math.Sqrt((108882498100.32983 / (3)) + (261581447394.0249 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.2824078512363567 = (70905026.88253969 - 90929163.16629383) / 70905026.88253969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: False, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -78.81117938606832 (T) = (0 -75638081.11874674) / Math.Sqrt((31282704720.40332 / (3)) + (403344443440.949 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.373256427440994 = (55079357.072222225 - 75638081.11874674) / 55079357.072222225 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: True, SkipValidation: False, DataSize: 10)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/28/2022 6:20:56 PM, 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -11.57590500424437 (T) = (0 -30231.31244313354) / Math.Sqrt((67666.7760823787 / (3)) + (117037.93763462499 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.08211676236427089 = (27937.200027363433 - 30231.31244313354) / 27937.200027363433 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Basic.WriteBasicUtf16(Formatted: False, SkipValidation: True, DataSize: 100000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -75.62578900760494 (T) = (0 -73020230.14722659) / Math.Sqrt((162112849368.19473 / (3)) + (146961748885.83612 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.39605175430297895 = (52304816.00854701 - 73020230.14722659) / 52304816.00854701 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TryFormat2 - Duration of single invocation 578.84 ns 974.11 ns 1.68 0.10 False
ToString4 - Duration of single invocation 1.42 μs 2.13 μs 1.50 0.08 False
ToString3 - Duration of single invocation 1.05 μs 1.67 μs 1.60 0.08 False
ToString2 - Duration of single invocation 797.23 ns 1.19 μs 1.50 0.09 False
ToStringL - Duration of single invocation 2.05 μs 2.85 μs 1.39 0.05 False
TryFormat4 - Duration of single invocation 1.12 μs 1.92 μs 1.72 0.06 False
TryFormatL - Duration of single invocation 1.74 μs 2.63 μs 1.51 0.06 False
TryFormat3 - Duration of single invocation 836.43 ns 1.43 μs 1.71 0.12 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Version.TryFormat2

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -41.08592443792672 (T) = (0 -968.2503671683143) / Math.Sqrt((163.074808044315 / (3)) + (275.2844502503585 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6969689673542422 = (570.5763545446097 - 968.2503671683143) / 570.5763545446097 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -22.928646975114198 (T) = (0 -2119.017697730877) / Math.Sqrt((3295.372207835757 / (3)) + (112.34313241532698 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5655114033419184 = (1353.5626078528596 - 2119.017697730877) / 1353.5626078528596 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -50.12121143510886 (T) = (0 -1670.3326233733162) / Math.Sqrt((128.46358579238984 / (3)) + (777.8228717528335 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5931167203568989 = (1048.4684530830352 - 1670.3326233733162) / 1048.4684530830352 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToString2

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -58.46214243246635 (T) = (0 -1187.8338880274496) / Math.Sqrt((84.27281257955545 / (3)) + (125.97271213687128 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.5017824102195412 = (790.949394495707 - 1187.8338880274496) / 790.949394495707 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.ToStringL

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -91.69194182665551 (T) = (0 -2857.211981289048) / Math.Sqrt((49.24079445585467 / (3)) + (415.8728392220872 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.38781380830399137 = (2058.78624653603 - 2857.211981289048) / 2058.78624653603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormat4

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -62.40308510854372 (T) = (0 -1889.2131903452803) / Math.Sqrt((286.5834475471491 / (3)) + (403.059532449967 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.6912401559315438 = (1117.057907902318 - 1889.2131903452803) / 1117.057907902318 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormatL

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -54.83040019686102 (T) = (0 -2607.6647343950035) / Math.Sqrt((34.681184609355796 / (3)) + (1646.6742331179755 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.49326236672907714 = (1746.2870507524901 - 2607.6647343950035) / 1746.2870507524901 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Tests.Perf_Version.TryFormat3

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -36.06921027761463 (T) = (0 -1450.508075229355) / Math.Sqrt((89.18504279424049 / (3)) + (1793.2760341180954 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.7255264823640271 = (840.6176839674532 - 1450.508075229355) / 840.6176839674532 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 99ba09612b31e90e1e7feb5511ee541f9f3ef02b
Compare e09e643ca91ab0deb427878a151875b2e5b7d53c
Diff Diff

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Append_Primitives - Duration of single invocation 25.08 μs 27.24 μs 1.09 0.04 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.Append_Primitives

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/14/2023 10:23:46 PM, 1/24/2023 1:05:59 AM falls between 1/14/2023 10:23:46 PM and 1/24/2023 1:05:59 AM.
IsRegressionStdDev: Marked as regression because -4.803488666766714 (T) = (0 -27426.272732450405) / Math.Sqrt((1151332.1502733005 / (3)) + (33399.24856288554 / (7))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (3) + (7) - 2, .025) and -0.12255161275101895 = (24432.081715367443 - 27426.272732450405) / 24432.081715367443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@kotlarmilos
Copy link
Member

Could be dotnet/runtime#80668. Should be fixed by dotnet/runtime#80762.

@kotlarmilos kotlarmilos mentioned this issue Jan 24, 2023
Closed
@kotlarmilos
Copy link
Member

Mitigated by dotnet/runtime#82093.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-interpreter os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos