Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 70 Regressions on 11/2/2022 4:25:40 PM #78813

Closed
performanceautofiler bot opened this issue Nov 15, 2022 · 6 comments
Closed

[Perf] Linux/x64: 70 Regressions on 11/2/2022 4:25:40 PM #78813

performanceautofiler bot opened this issue Nov 15, 2022 · 6 comments
Assignees
@tannergooding
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 16.84 ns 40.53 ns 2.41 0.29 False
InequalityOperatorBenchmark - Duration of single invocation 20.72 ns 187.11 ns 9.03 0.18 False
XorBenchmark - Duration of single invocation 17.25 ns 40.81 ns 2.37 0.30 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.99 ns 42.92 ns 2.53 0.24 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 16.96 ns 40.42 ns 2.38 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.5291062465591 > 18.477594878563934.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -47.239308415870234 (T) = (0 -41.41790920141574) / Math.Sqrt((3.335561473944583 / (20)) + (2.1633274825376514 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.290622176840343 = (18.08151061321999 - 41.41790920141574) / 18.08151061321999 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 187.11408029229804 > 21.764436301044054.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -431.1909578423119 (T) = (0 -188.47035820009683) / Math.Sqrt((1.425716122197386 / (20)) + (2.142805041765067 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -7.928031305937372 = (21.10995713856415 - 188.47035820009683) / 21.10995713856415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.81021480107198 > 18.116704313226663.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -127.68319021486 (T) = (0 -41.193711323552684) / Math.Sqrt((0.2684536573634024 / (20)) + (0.5997029688187765 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.372798251038393 = (17.360814938870313 - 41.193711323552684) / 17.360814938870313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.91524687261434 > 17.394458809752205.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -93.67865366396816 (T) = (0 -41.746677469431766) / Math.Sqrt((0.17661552510191394 / (20)) + (1.7197998782803212 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4677313303020454 = (16.917026969999224 - 41.746677469431766) / 16.917026969999224 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.419495762951534 > 18.591036246729598.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -83.1982748541115 (T) = (0 -40.85444455175857) / Math.Sqrt((0.18369627136157285 / (20)) + (1.9823321347648852 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3581196749738356 = (17.32500898293547 - 40.85444455175857) / 17.32500898293547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Widen_int - Duration of single invocation 141.13 μs 202.45 μs 1.43 0.15 False
Widen_sbyte - Duration of single invocation 254.78 μs 293.15 μs 1.15 0.16 False
Widen_short - Duration of single invocation 173.14 μs 251.39 μs 1.45 0.15 False
Widen_byte - Duration of single invocation 237.03 μs 293.97 μs 1.24 0.13 False
Widen_ushort - Duration of single invocation 174.06 μs 238.15 μs 1.37 0.15 False
Convert_float_uint - Duration of single invocation 80.85 μs 111.35 μs 1.38 0.16 False
Narrow_ulong - Duration of single invocation 77.24 μs 103.75 μs 1.34 0.20 False
Convert_int_float - Duration of single invocation 70.65 μs 99.47 μs 1.41 0.24 False
Widen_float - Duration of single invocation 141.10 μs 204.83 μs 1.45 0.17 False
Widen_uint - Duration of single invocation 137.81 μs 214.77 μs 1.56 0.16 False
Narrow_long - Duration of single invocation 76.62 μs 102.91 μs 1.34 0.23 False
Convert_double_ulong - Duration of single invocation 51.04 μs 77.66 μs 1.52 0.19 False
Convert_double_long - Duration of single invocation 48.66 μs 75.07 μs 1.54 0.22 False
Convert_float_int - Duration of single invocation 69.69 μs 92.30 μs 1.32 0.20 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorConvert.Widen_int

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 202.45187283755277 > 150.53291864218883.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -44.392517453466986 (T) = (0 -205205.4717936811) / Math.Sqrt((17047800.665222235 / (20)) + (26254225.7519758 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.4398249101984699 = (142521.12902074665 - 205205.4717936811) / 142521.12902074665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 293.15014382861636 > 253.48416977982956.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -35.2733146379764 (T) = (0 -296947.73934133037) / Math.Sqrt((25800730.454728354 / (20)) + (28667621.617200498 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.23332240111072455 = (240770.57148552607 - 296947.73934133037) / 240770.57148552607 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_short

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 251.39353301411293 > 187.19400435014205.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -35.92446333551712 (T) = (0 -237528.95567246663) / Math.Sqrt((20242403.494790737 / (20)) + (40514832.45087705 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.33668879438614013 = (177699.51889328827 - 237528.95567246663) / 177699.51889328827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_byte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 293.9710959821428 > 251.65250061789772.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -23.063582348840256 (T) = (0 -298220.0057950873) / Math.Sqrt((102133330.83930433 / (20)) + (23121979.477138925 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.23639469786909909 = (241201.29786148662 - 298220.0057950873) / 241201.29786148662 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 238.1533576388889 > 187.63593941210323.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -38.088195935097346 (T) = (0 -237566.33403132632) / Math.Sqrt((31928429.86213346 / (20)) + (17086500.39751962 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3248831955340466 = (179311.15349045227 - 237566.33403132632) / 179311.15349045227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 111.35233658134906 > 86.08000324995476.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -44.749915078571014 (T) = (0 -112810.09268030038) / Math.Sqrt((6929127.866056547 / (20)) + (3387118.412275842 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.3721308958171664 = (82215.25586530638 - 112810.09268030038) / 82215.25586530638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 103.75428016552513 > 84.12624158655468.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -16.15209807407399 (T) = (0 -107176.0917190748) / Math.Sqrt((27835080.056711826 / (20)) + (30554517.437756278 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3307793596113731 = (80536.33455088557 - 107176.0917190748) / 80536.33455088557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 99.4732965848591 > 76.33051442006912.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -33.12063394975481 (T) = (0 -99295.60720210151) / Math.Sqrt((2985805.5791938826 / (20)) + (14627038.325555101 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.37621672824097874 = (72151.14099725914 - 99295.60720210151) / 72151.14099725914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 204.83184479166664 > 153.1866366134914.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -46.58694895639269 (T) = (0 -205761.08831465658) / Math.Sqrt((22299698.11055435 / (20)) + (12728293.43090577 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.4133250564365632 = (145586.52829197337 - 205761.08831465658) / 145586.52829197337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 214.77136901709403 > 141.8052844127155.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -70.04167567523523 (T) = (0 -215527.59608299905) / Math.Sqrt((15624936.986918148 / (20)) + (17251378.577549364 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.6237909747153918 = (132731.12083947592 - 215527.59608299905) / 132731.12083947592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 102.90969040765978 > 82.54550129498229.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -26.948783764420856 (T) = (0 -104362.5909463695) / Math.Sqrt((14867113.450207755 / (20)) + (3069513.231509969 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3189049384052961 = (79128.21304055133 - 104362.5909463695) / 79128.21304055133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 77.65547291945028 > 54.68652949553188.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -43.757850376123564 (T) = (0 -78967.20161528121) / Math.Sqrt((3291410.29939871 / (20)) + (6043420.727478606 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.5192249398478216 = (51978.61063496709 - 78967.20161528121) / 51978.61063496709 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 75.07440263835264 > 51.69023774659832.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -49.92012650273555 (T) = (0 -76422.80895744797) / Math.Sqrt((86066.37623833306 / (20)) + (8335228.1421386385 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.5599709558650328 = (48989.89219646728 - 76422.80895744797) / 48989.89219646728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 92.29873638146115 > 74.42708289638192.
IsChangePoint: Marked as a change because one of 9/29/2022 2:35:34 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -15.05640523470537 (T) = (0 -95089.49746209107) / Math.Sqrt((19324258.958696887 / (19)) + (36988041.5296486 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.31945711708042124 = (72067.13748492012 - 95089.49746209107) / 72067.13748492012 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 22.11 ns 101.50 ns 4.59 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 16.82 ns 40.32 ns 2.40 0.26 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.79 ns 40.54 ns 2.41 0.29 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.81 ns 40.47 ns 2.27 0.28 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 101.49724760165354 > 23.64783511657208.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -106.98125356783797 (T) = (0 -103.03808231890699) / Math.Sqrt((0.40070611311124504 / (20)) + (15.251270093477986 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -3.550405366791448 = (22.64371501292434 - 103.03808231890699) / 22.64371501292434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.31880881310092 > 17.83128627485227.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -42.61298545080504 (T) = (0 -41.89209770719573) / Math.Sqrt((2.3313680079111316 / (20)) + (6.069016654476517 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4229079108055214 = (17.29000822539238 - 41.89209770719573) / 17.29000822539238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.53559265574434 > 17.548768789832028.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -59.007769315837415 (T) = (0 -41.80813215397197) / Math.Sqrt((1.2463368406120203 / (20)) + (3.0989098570753373 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -1.4627271310684615 = (16.97635585629553 - 41.80813215397197) / 16.97635585629553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.46596674062233 > 18.495657668558.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -70.4654635650013 (T) = (0 -41.15400857771605) / Math.Sqrt((1.7863782882589108 / (20)) + (0.6810235480348938 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3652270280782786 = (17.39960185181599 - 41.15400857771605) / 17.39960185181599 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseOrOperatorBenchmark - Duration of single invocation 16.66 ns 40.53 ns 2.43 0.24 False
BitwiseAndOperatorBenchmark - Duration of single invocation 17.74 ns 40.62 ns 2.29 0.28 False
BitwiseAndBenchmark - Duration of single invocation 16.76 ns 40.81 ns 2.43 0.32 False
BitwiseOrBenchmark - Duration of single invocation 16.57 ns 41.02 ns 2.48 0.28 False
AndNotBenchmark - Duration of single invocation 46.26 ns 67.32 ns 1.46 0.21 False
ConditionalSelectBenchmark - Duration of single invocation 69.21 ns 161.16 ns 2.33 0.29 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt64>.BitwiseOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.53164799814567 > 17.606145519706605.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -128.38139613997168 (T) = (0 -41.20890879512215) / Math.Sqrt((0.34195441854849323 / (20)) + (0.5157358600305211 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4220221761530267 = (17.014257425411166 - 41.20890879512215) / 17.014257425411166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.62477765107687 > 18.556256515179886.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -51.00414445642825 (T) = (0 -41.79906614593597) / Math.Sqrt((1.1860031427503732 / (20)) + (4.59187564049868 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3618657046392864 = (17.697477914951854 - 41.79906614593597) / 17.697477914951854 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.806000323059074 > 18.000723088789336.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -52.56742683470581 (T) = (0 -41.535895628561136) / Math.Sqrt((0.9109152610869186 / (20)) + (3.9727642880123186 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.446675755970931 = (16.976461031747203 - 41.535895628561136) / 16.976461031747203 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.024393043986876 > 17.647090705916188.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -29.29893245071074 (T) = (0 -42.56984368137655) / Math.Sqrt((0.5477751696249815 / (20)) + (16.655209742119613 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.479259781126249 = (17.170384485500918 - 42.56984368137655) / 17.170384485500918 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 67.31684977288064 > 49.59935749463632.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -25.154157997429106 (T) = (0 -69.03710127695737) / Math.Sqrt((9.319261052346013 / (20)) + (5.589499740825489 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.442570907647478 = (47.85698984429264 - 69.03710127695737) / 47.85698984429264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 161.1594956652279 > 74.0089322996093.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -78.32175482055422 (T) = (0 -162.16007864380046) / Math.Sqrt((10.982887565034181 / (20)) + (18.263986827367287 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.2715675328309755 = (71.38686228786956 - 162.16007864380046) / 71.38686228786956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Double>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.80 ns 41.95 ns 2.36 0.28 False
AndNotBenchmark - Duration of single invocation 47.66 ns 68.74 ns 1.44 0.25 False
ConditionalSelectBenchmark - Duration of single invocation 72.06 ns 156.72 ns 2.18 0.21 False
InequalityOperatorBenchmark - Duration of single invocation 23.79 ns 38.78 ns 1.63 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 17.57 ns 40.67 ns 2.31 0.25 False
XorBenchmark - Duration of single invocation 16.70 ns 40.41 ns 2.42 0.27 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.76 ns 51.63 ns 3.08 0.23 False
BitwiseAndBenchmark - Duration of single invocation 16.78 ns 40.40 ns 2.41 0.26 False
BitwiseOrBenchmark - Duration of single invocation 17.20 ns 41.05 ns 2.39 0.24 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Double>.ExclusiveOrOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.95414894804784 > 18.142174583884447.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -95.70810330271725 (T) = (0 -41.399319416075194) / Math.Sqrt((0.31438746164885384 / (20)) + (1.105665083060244 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.403263415139443 = (17.22629286297903 - 41.399319416075194) / 17.22629286297903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 68.73812295950513 > 50.57955033518685.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -27.919404305716228 (T) = (0 -69.27600731919406) / Math.Sqrt((2.575919161711604 / (20)) + (11.802677886827691 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.4264772411720593 = (48.56439718748945 - 69.27600731919406) / 48.56439718748945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 156.72379328259157 > 75.60379808075723.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -73.3740206275856 (T) = (0 -165.06895531493265) / Math.Sqrt((4.059110572266824 / (19)) + (38.50399276371836 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -1.2741990050817324 = (72.58333810985035 - 165.06895531493265) / 72.58333810985035 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 38.782002692567644 > 25.87598115980782.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -29.887801194339755 (T) = (0 -38.96978317551664) / Math.Sqrt((4.153596123527951 / (20)) + (0.32169782783013284 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.5651671228002333 = (24.898161102308347 - 38.96978317551664) / 24.898161102308347 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.66889798735548 > 18.094842072275636.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -51.90244996519183 (T) = (0 -40.887704538977154) / Math.Sqrt((3.8084911524230014 / (20)) + (0.16421779607422773 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.2947431619574086 = (17.817987309786805 - 40.887704538977154) / 17.817987309786805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.40770570584669 > 18.32760320209184.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -61.61580008253972 (T) = (0 -41.502934354725845) / Math.Sqrt((2.045347269011827 / (19)) + (1.1856477816221982 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -1.3528165412918172 = (17.639681473821415 - 41.502934354725845) / 17.639681473821415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 51.629082560299366 > 17.717786952501772.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -38.79609409108265 (T) = (0 -42.6374944304798) / Math.Sqrt((0.12539989529380435 / (20)) + (9.930268399510137 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.5138518183849548 = (16.96102137709637 - 42.6374944304798) / 16.96102137709637 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.40317101463805 > 18.030950156854956.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -158.15331793494605 (T) = (0 -41.04859905334506) / Math.Sqrt((0.2741195848128902 / (20)) + (0.24383083808857148 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -1.3863913269481178 = (17.201118102386353 - 41.04859905334506) / 17.201118102386353 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Double&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.05049243376845 > 17.955748589529485.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -64.81207181293068 (T) = (0 -41.888901837093826) / Math.Sqrt((0.47394292749865446 / (20)) + (3.414475702639503 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4418516331031153 = (17.15456470377819 - 41.888901837093826) / 17.15456470377819 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int64>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 16.72 ns 40.20 ns 2.40 0.31 False
AndNotBenchmark - Duration of single invocation 52.29 ns 68.07 ns 1.30 0.24 False
BitwiseOrBenchmark - Duration of single invocation 25.56 ns 40.72 ns 1.59 0.28 False
BitwiseAndBenchmark - Duration of single invocation 16.50 ns 41.66 ns 2.52 0.29 False
ConditionalSelectBenchmark - Duration of single invocation 71.55 ns 167.41 ns 2.34 0.22 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int64>.XorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.20036817215005 > 17.85565751722261.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -53.71257389076695 (T) = (0 -41.3842009008731) / Math.Sqrt((3.025801694366913 / (20)) + (0.9823825532116779 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.334644972788616 = (17.726121694401247 - 41.3842009008731) / 17.726121694401247 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.AndNotBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 68.0703192114721 > 49.770646070928855.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -25.98681790783077 (T) = (0 -68.76970570747753) / Math.Sqrt((7.817606005051727 / (20)) + (5.665810740494327 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.43194284646244663 = (48.02545428218042 - 68.76970570747753) / 48.02545428218042 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.715323379085895 > 17.942548458246744.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -41.53879900450213 (T) = (0 -41.27653404371267) / Math.Sqrt((6.062666140522287 / (20)) + (0.4210584938699945 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.32861034014898 = (17.725822707234 - 41.27653404371267) / 17.725822707234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.66222605547739 > 18.066355478075902.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -95.18517402448734 (T) = (0 -41.42674139088859) / Math.Sqrt((0.39837945120547785 / (20)) + (1.0209891906534465 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.3962197833328494 = (17.288372994429185 - 41.42674139088859) / 17.288372994429185 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int64&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 167.41362945855627 > 74.17101663385678.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -90.55532413110294 (T) = (0 -162.438653152912) / Math.Sqrt((9.630287964863703 / (20)) + (14.81593363001358 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.2749817365044953 = (71.40217899177452 - 162.438653152912) / 71.40217899177452 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 25.08 ns 56.91 ns 2.27 0.20 False
BitwiseAndOperatorBenchmark - Duration of single invocation 16.74 ns 43.12 ns 2.58 0.24 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 16.97 ns 49.86 ns 2.94 0.26 False
BitwiseAndBenchmark - Duration of single invocation 16.44 ns 40.53 ns 2.47 0.25 False
XorBenchmark - Duration of single invocation 16.71 ns 40.51 ns 2.42 0.24 False
ConditionalSelectBenchmark - Duration of single invocation 80.38 ns 157.20 ns 1.96 0.25 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.81 ns 40.76 ns 2.42 0.27 False
BitwiseOrBenchmark - Duration of single invocation 17.59 ns 41.24 ns 2.34 0.29 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int32>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 56.90826973565509 > 24.247221074568618.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -99.59423738112983 (T) = (0 -57.61943196494066) / Math.Sqrt((1.0798305682111415 / (20)) + (1.890886762303809 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.516055062064843 = (22.90070389701818 - 57.61943196494066) / 22.90070389701818 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 43.12137498424083 > 18.11429683998205.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -43.807659692402225 (T) = (0 -41.3767985651958) / Math.Sqrt((1.9618601619454 / (20)) + (4.570680905472447 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.363047052353541 = (17.50993427066357 - 41.3767985651958) / 17.50993427066357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 49.85572505132034 > 18.138536675707396.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -54.60078210309482 (T) = (0 -41.78011117782496) / Math.Sqrt((0.10870220987329464 / (20)) + (5.559985772105357 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4406321567680367 = (17.11856129649276 - 41.78011117782496) / 17.11856129649276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.53343999470511 > 17.7965364809841.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -30.230065138454485 (T) = (0 -42.95344137596823) / Math.Sqrt((3.6624410496679154 / (20)) + (14.128836175060828 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4017905915776538 = (17.88392440481399 - 42.95344137596823) / 17.88392440481399 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.511991454449664 > 18.034757911519705.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -53.35195153252286 (T) = (0 -41.35633638022165) / Math.Sqrt((1.7130634948143286 / (20)) + (3.2063789028740644 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.365019822745808 = (17.486676425488305 - 41.35633638022165) / 17.486676425488305 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 157.20238805211756 > 87.23111251102775.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -43.0739953212709 (T) = (0 -163.8608759156622) / Math.Sqrt((35.16668767283196 / (20)) + (49.26781715312335 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.9725396966415338 = (83.0710156021972 - 163.8608759156622) / 83.0710156021972 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.75689313255259 > 17.67608491781333.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -57.11487856553796 (T) = (0 -42.120454947687165) / Math.Sqrt((0.1882326419587469 / (20)) + (4.21749519510436 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.4714049929001631 = (17.04312124831444 - 42.120454947687165) / 17.04312124831444 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.235087890359395 > 17.8663204261076.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -48.55980730552291 (T) = (0 -41.22411091840453) / Math.Sqrt((4.517193528452578 / (20)) + (0.27910360491125863 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3366427549031268 = (17.642453401103502 - 41.22411091840453) / 17.642453401103502 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Single>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndBenchmark - Duration of single invocation 16.76 ns 40.62 ns 2.42 0.30 False
XorBenchmark - Duration of single invocation 17.15 ns 40.43 ns 2.36 0.23 False
ConditionalSelectBenchmark - Duration of single invocation 83.32 ns 157.15 ns 1.89 0.25 False
InequalityOperatorBenchmark - Duration of single invocation 23.85 ns 66.13 ns 2.77 0.18 False
BitwiseOrBenchmark - Duration of single invocation 17.99 ns 41.07 ns 2.28 0.27 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.77 ns 41.96 ns 2.36 0.31 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Single>.BitwiseAndBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.62206714582384 > 18.27329361617259.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -47.739918677414266 (T) = (0 -41.479927603809145) / Math.Sqrt((2.3556206606376513 / (20)) + (3.775837769730324 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3723655113195554 = (17.48462764523044 - 41.479927603809145) / 17.48462764523044 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.43210440749699 > 17.949384918494065.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -45.02591891724047 (T) = (0 -41.611464280960206) / Math.Sqrt((2.1008415408761634 / (20)) + (4.224144404970851 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.3889028192601787 = (17.418650916007916 - 41.611464280960206) / 17.418650916007916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 157.151778688581 > 87.29918799615037.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -44.27925286615416 (T) = (0 -163.25781854025144) / Math.Sqrt((33.878573912797705 / (19)) + (37.45719648915953 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.9199001086002287 = (85.0345378954535 - 163.25781854025144) / 85.0345378954535 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 66.12759452186134 > 25.233368805216987.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -97.54556876821384 (T) = (0 -66.72242967905336) / Math.Sqrt((1.0202784483533542 / (20)) + (3.1370577643381337 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.7239923676083524 = (24.494352654018346 - 66.72242967905336) / 24.494352654018346 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.06821154203491 > 17.96666543171592.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -62.517638511074345 (T) = (0 -41.71164139390404) / Math.Sqrt((0.4476425286157613 / (20)) + (3.701305213254657 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4346556933118964 = (17.13246004701515 - 41.71164139390404) / 17.13246004701515 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Single&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.95911752537822 > 18.290356882863918.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -86.25777280610437 (T) = (0 -41.36677368043005) / Math.Sqrt((0.552194104434982 / (20)) + (1.1506621745714107 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.3867035276189803 = (17.332179385387807 - 41.36677368043005) / 17.332179385387807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 17.21 ns 40.55 ns 2.36 0.28 False
BitwiseOrOperatorBenchmark - Duration of single invocation 18.26 ns 41.04 ns 2.25 0.23 False
BitwiseAndBenchmark - Duration of single invocation 16.91 ns 40.63 ns 2.40 0.24 False
ConditionalSelectBenchmark - Duration of single invocation 101.83 ns 162.44 ns 1.60 0.21 False
BitwiseOrBenchmark - Duration of single invocation 17.83 ns 41.50 ns 2.33 0.30 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<Int16>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.55038171922145 > 18.084935946084464.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -35.299614948005065 (T) = (0 -42.47716770211326) / Math.Sqrt((2.2440068723989772 / (20)) + (10.716849520261919 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4075840086520082 = (17.64306771828742 - 42.47716770211326) / 17.64306771828742 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.036896224165396 > 18.012634564461163.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -58.34419764641006 (T) = (0 -42.17210183093418) / Math.Sqrt((0.4993148876985614 / (20)) + (4.379863786766525 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4344008144284839 = (17.323401134679123 - 42.17210183093418) / 17.323401134679123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.62513284137419 > 18.123232739825486.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -38.330409789434604 (T) = (0 -42.19023698233719) / Math.Sqrt((0.8616236647963296 / (20)) + (8.678921582592798 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.433578134813372 = (17.336709423374526 - 42.19023698233719) / 17.336709423374526 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 162.4428813729129 > 107.58919212383904.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -54.78559747026722 (T) = (0 -161.77688617570277) / Math.Sqrt((9.903955811011883 / (20)) + (14.618954922559226 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.5627858502538504 = (103.51826908941146 - 161.77688617570277) / 103.51826908941146 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;Int16&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.49665226919189 > 18.362462858678224.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -45.61974975855345 (T) = (0 -41.90277825140024) / Math.Sqrt((4.701935252431349 / (20)) + (1.2828145392804862 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.4222781129218296 = (17.298912964562838 - 41.90277825140024) / 17.298912964562838 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
XorBenchmark - Duration of single invocation 16.63 ns 41.00 ns 2.47 0.24 False
ConditionalSelectBenchmark - Duration of single invocation 80.01 ns 156.66 ns 1.96 0.24 False
InequalityOperatorBenchmark - Duration of single invocation 22.42 ns 60.30 ns 2.69 0.21 False
BitwiseOrBenchmark - Duration of single invocation 15.80 ns 41.21 ns 2.61 0.25 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.78 ns 40.98 ns 2.44 0.24 False
BitwiseAndOperatorBenchmark - Duration of single invocation 17.27 ns 40.90 ns 2.37 0.19 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.07 ns 40.49 ns 2.37 0.24 False
BitwiseAndBenchmark - Duration of single invocation 17.02 ns 41.30 ns 2.43 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt32>.XorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.00418918348813 > 18.542394021193637.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -50.249974383968215 (T) = (0 -41.07894911281636) / Math.Sqrt((3.295983487944851 / (20)) + (1.3425042321552285 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.2947255725571813 = (17.901464821799614 - 41.07894911281636) / 17.901464821799614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ConditionalSelectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 156.65656393913815 > 85.59147409888784.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -90.75649606749998 (T) = (0 -162.0977866786523) / Math.Sqrt((4.831538779053969 / (20)) + (15.100935285086852 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.9792692953045857 = (81.89779281838827 - 162.0977866786523) / 81.89779281838827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 60.30170146687125 > 23.763736308727378.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -89.80752978769853 (T) = (0 -57.50800672598785) / Math.Sqrt((1.9571400981551965 / (20)) + (1.1545459615286833 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.5055791908954441 = (22.95198129636271 - 57.50800672598785) / 22.95198129636271 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseOrBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.21239857790485 > 17.69870802188416.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -105.72718445177621 (T) = (0 -41.617528318888326) / Math.Sqrt((0.1784409012685139 / (20)) + (1.2256110964509384 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -1.4513797862869358 = (16.97718507417641 - 41.617528318888326) / 16.97718507417641 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.98067706693471 > 17.782799807251045.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -113.3630334772679 (T) = (0 -41.48820983761845) / Math.Sqrt((0.2981103630398961 / (20)) + (0.7188883518906913 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.4216940914853622 = (17.131895388228568 - 41.48820983761845) / 17.131895388228568 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.89842467280184 > 18.044617772391657.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -55.31896731276874 (T) = (0 -41.741306050362844) / Math.Sqrt((0.7961318052241168 / (20)) + (3.5651333426806273 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.4093292714123495 = (17.324865698366782 - 41.741306050362844) / 17.324865698366782 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.4852851734626 > 17.990926721021495.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -68.57535820610904 (T) = (0 -41.102927124477986) / Math.Sqrt((1.1907101826961894 / (20)) + (1.399937931587245 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -1.3748028472267988 = (17.307932392146306 - 41.102927124477986) / 17.307932392146306 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt32&gt;.BitwiseAndBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 41.2962777253215 > 17.998051597130623.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -57.02666023051669 (T) = (0 -41.475997488202374) / Math.Sqrt((1.828045554148332 / (20)) + (2.4449167511394942 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3881024528721637 = (17.367763028055734 - 41.475997488202374) / 17.367763028055734 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added PGO refs/heads/main untriaged New issue has not been triaged by the area owner labels Nov 15, 2022
@kotlarmilos
Copy link
Member

Looks like regression in 13d4e6d

@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 16, 2022
@kotlarmilos kotlarmilos mentioned this issue Nov 16, 2022
Closed
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Nov 24, 2022
@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@kotlarmilos kotlarmilos removed the untriaged New issue has not been triaged by the area owner label Nov 24, 2022
@jeffschwMSFT jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Nov 25, 2022
@ghost
Copy link

ghost commented Nov 25, 2022

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<SByte>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitwiseAndOperatorBenchmark - Duration of single invocation 16.84 ns 40.53 ns 2.41 0.29 False
InequalityOperatorBenchmark - Duration of single invocation 20.72 ns 187.11 ns 9.03 0.18 False
XorBenchmark - Duration of single invocation 17.25 ns 40.81 ns 2.37 0.30 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.99 ns 42.92 ns 2.53 0.24 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 16.96 ns 40.42 ns 2.38 0.26 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<SByte>.BitwiseAndOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.5291062465591 > 18.477594878563934.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -47.239308415870234 (T) = (0 -41.41790920141574) / Math.Sqrt((3.335561473944583 / (20)) + (2.1633274825376514 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.290622176840343 = (18.08151061321999 - 41.41790920141574) / 18.08151061321999 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.InequalityOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 187.11408029229804 > 21.764436301044054.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -431.1909578423119 (T) = (0 -188.47035820009683) / Math.Sqrt((1.425716122197386 / (20)) + (2.142805041765067 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -7.928031305937372 = (21.10995713856415 - 188.47035820009683) / 21.10995713856415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.XorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.81021480107198 > 18.116704313226663.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -127.68319021486 (T) = (0 -41.193711323552684) / Math.Sqrt((0.2684536573634024 / (20)) + (0.5997029688187765 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.372798251038393 = (17.360814938870313 - 41.193711323552684) / 17.360814938870313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 42.91524687261434 > 17.394458809752205.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -93.67865366396816 (T) = (0 -41.746677469431766) / Math.Sqrt((0.17661552510191394 / (20)) + (1.7197998782803212 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4677313303020454 = (16.917026969999224 - 41.746677469431766) / 16.917026969999224 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;SByte&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.419495762951534 > 18.591036246729598.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -83.1982748541115 (T) = (0 -40.85444455175857) / Math.Sqrt((0.18369627136157285 / (20)) + (1.9823321347648852 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3581196749738356 = (17.32500898293547 - 40.85444455175857) / 17.32500898293547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare 803688595257517f4b47e873d11152b0560c204a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Widen_int - Duration of single invocation 141.13 μs 202.45 μs 1.43 0.15 False
Widen_sbyte - Duration of single invocation 254.78 μs 293.15 μs 1.15 0.16 False
Widen_short - Duration of single invocation 173.14 μs 251.39 μs 1.45 0.15 False
Widen_byte - Duration of single invocation 237.03 μs 293.97 μs 1.24 0.13 False
Widen_ushort - Duration of single invocation 174.06 μs 238.15 μs 1.37 0.15 False
Convert_float_uint - Duration of single invocation 80.85 μs 111.35 μs 1.38 0.16 False
Narrow_ulong - Duration of single invocation 77.24 μs 103.75 μs 1.34 0.20 False
Convert_int_float - Duration of single invocation 70.65 μs 99.47 μs 1.41 0.24 False
Widen_float - Duration of single invocation 141.10 μs 204.83 μs 1.45 0.17 False
Widen_uint - Duration of single invocation 137.81 μs 214.77 μs 1.56 0.16 False
Narrow_long - Duration of single invocation 76.62 μs 102.91 μs 1.34 0.23 False
Convert_double_ulong - Duration of single invocation 51.04 μs 77.66 μs 1.52 0.19 False
Convert_double_long - Duration of single invocation 48.66 μs 75.07 μs 1.54 0.22 False
Convert_float_int - Duration of single invocation 69.69 μs 92.30 μs 1.32 0.20 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorConvert*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorConvert.Widen_int

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 202.45187283755277 > 150.53291864218883.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -44.392517453466986 (T) = (0 -205205.4717936811) / Math.Sqrt((17047800.665222235 / (20)) + (26254225.7519758 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.4398249101984699 = (142521.12902074665 - 205205.4717936811) / 142521.12902074665 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_sbyte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 293.15014382861636 > 253.48416977982956.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -35.2733146379764 (T) = (0 -296947.73934133037) / Math.Sqrt((25800730.454728354 / (20)) + (28667621.617200498 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.23332240111072455 = (240770.57148552607 - 296947.73934133037) / 240770.57148552607 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_short

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 251.39353301411293 > 187.19400435014205.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -35.92446333551712 (T) = (0 -237528.95567246663) / Math.Sqrt((20242403.494790737 / (20)) + (40514832.45087705 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.33668879438614013 = (177699.51889328827 - 237528.95567246663) / 177699.51889328827 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_byte

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 293.9710959821428 > 251.65250061789772.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -23.063582348840256 (T) = (0 -298220.0057950873) / Math.Sqrt((102133330.83930433 / (20)) + (23121979.477138925 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.23639469786909909 = (241201.29786148662 - 298220.0057950873) / 241201.29786148662 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_ushort

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 238.1533576388889 > 187.63593941210323.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -38.088195935097346 (T) = (0 -237566.33403132632) / Math.Sqrt((31928429.86213346 / (20)) + (17086500.39751962 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3248831955340466 = (179311.15349045227 - 237566.33403132632) / 179311.15349045227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 111.35233658134906 > 86.08000324995476.
IsChangePoint: Marked as a change because one of 9/28/2022 3:03:11 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -44.749915078571014 (T) = (0 -112810.09268030038) / Math.Sqrt((6929127.866056547 / (20)) + (3387118.412275842 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.3721308958171664 = (82215.25586530638 - 112810.09268030038) / 82215.25586530638 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 103.75428016552513 > 84.12624158655468.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -16.15209807407399 (T) = (0 -107176.0917190748) / Math.Sqrt((27835080.056711826 / (20)) + (30554517.437756278 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3307793596113731 = (80536.33455088557 - 107176.0917190748) / 80536.33455088557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_int_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 99.4732965848591 > 76.33051442006912.
IsChangePoint: Marked as a change because one of 9/28/2022 6:40:13 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -33.12063394975481 (T) = (0 -99295.60720210151) / Math.Sqrt((2985805.5791938826 / (20)) + (14627038.325555101 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.37621672824097874 = (72151.14099725914 - 99295.60720210151) / 72151.14099725914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_float

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 204.83184479166664 > 153.1866366134914.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -46.58694895639269 (T) = (0 -205761.08831465658) / Math.Sqrt((22299698.11055435 / (20)) + (12728293.43090577 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.4133250564365632 = (145586.52829197337 - 205761.08831465658) / 145586.52829197337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Widen_uint

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 214.77136901709403 > 141.8052844127155.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -70.04167567523523 (T) = (0 -215527.59608299905) / Math.Sqrt((15624936.986918148 / (20)) + (17251378.577549364 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.6237909747153918 = (132731.12083947592 - 215527.59608299905) / 132731.12083947592 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Narrow_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 102.90969040765978 > 82.54550129498229.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -26.948783764420856 (T) = (0 -104362.5909463695) / Math.Sqrt((14867113.450207755 / (20)) + (3069513.231509969 / (23))) is less than -2.019540970439573 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (23) - 2, .025) and -0.3189049384052961 = (79128.21304055133 - 104362.5909463695) / 79128.21304055133 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_ulong

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 77.65547291945028 > 54.68652949553188.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -43.757850376123564 (T) = (0 -78967.20161528121) / Math.Sqrt((3291410.29939871 / (20)) + (6043420.727478606 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.5192249398478216 = (51978.61063496709 - 78967.20161528121) / 51978.61063496709 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_double_long

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 75.07440263835264 > 51.69023774659832.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -49.92012650273555 (T) = (0 -76422.80895744797) / Math.Sqrt((86066.37623833306 / (20)) + (8335228.1421386385 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -0.5599709558650328 = (48989.89219646728 - 76422.80895744797) / 48989.89219646728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorConvert.Convert_float_int

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 92.29873638146115 > 74.42708289638192.
IsChangePoint: Marked as a change because one of 9/29/2022 2:35:34 PM, 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -15.05640523470537 (T) = (0 -95089.49746209107) / Math.Sqrt((19324258.958696887 / (19)) + (36988041.5296486 / (28))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (19) + (28) - 2, .025) and -0.31945711708042124 = (72067.13748492012 - 95089.49746209107) / 72067.13748492012 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline bdd67af67dbf43c0f5e9151b98dc100b9555898a
Compare fbc20578253e0deb7267eea5a02b680c3cfe0e2a
Diff Diff

Regressions in System.Numerics.Tests.Perf_VectorOf<UInt16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
InequalityOperatorBenchmark - Duration of single invocation 22.11 ns 101.50 ns 4.59 0.22 False
BitwiseAndOperatorBenchmark - Duration of single invocation 16.82 ns 40.32 ns 2.40 0.26 False
BitwiseOrOperatorBenchmark - Duration of single invocation 16.79 ns 40.54 ns 2.41 0.29 False
ExclusiveOrOperatorBenchmark - Duration of single invocation 17.81 ns 40.47 ns 2.27 0.28 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Numerics.Tests.Perf_VectorOf<UInt16>.InequalityOperatorBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 101.49724760165354 > 23.64783511657208.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -106.98125356783797 (T) = (0 -103.03808231890699) / Math.Sqrt((0.40070611311124504 / (20)) + (15.251270093477986 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -3.550405366791448 = (22.64371501292434 - 103.03808231890699) / 22.64371501292434 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseAndOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.31880881310092 > 17.83128627485227.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -42.61298545080504 (T) = (0 -41.89209770719573) / Math.Sqrt((2.3313680079111316 / (20)) + (6.069016654476517 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.4229079108055214 = (17.29000822539238 - 41.89209770719573) / 17.29000822539238 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.BitwiseOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.53559265574434 > 17.548768789832028.
IsChangePoint: Marked as a change because one of 10/10/2022 9:53:50 PM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -59.007769315837415 (T) = (0 -41.80813215397197) / Math.Sqrt((1.2463368406120203 / (20)) + (3.0989098570753373 / (27))) is less than -2.0141033888794695 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (27) - 2, .025) and -1.4627271310684615 = (16.97635585629553 - 41.80813215397197) / 16.97635585629553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_VectorOf&lt;UInt16&gt;.ExclusiveOrOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 40.46596674062233 > 18.495657668558.
IsChangePoint: Marked as a change because one of 10/11/2022 6:23:40 AM, 11/2/2022 9:35:40 AM, 11/9/2022 12:33:05 PM falls between 10/31/2022 10:23:25 PM and 11/9/2022 12:33:05 PM.
IsRegressionStdDev: Marked as regression because -70.4654635650013 (T) = (0 -41.15400857771605) / Math.Sqrt((1.7863782882589108 / (20)) + (0.6810235480348938 / (28))) is less than -2.0128955989180297 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (28) - 2, .025) and -1.3652270280782786 = (17.39960185181599 - 41.15400857771605) / 17.39960185181599 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: tannergooding
Labels:

area-CodeGen-coreclr, refs/heads/main, ubuntu 18.04, PGO, Regression, x64
Milestone: -

@JulieLeeMSFT JulieLeeMSFT added this to the 8.0.0 milestone Nov 26, 2022
@kotlarmilos
Copy link
Member

Fixed by b77aa8a

@jeffhandley jeffhandley added os-linux Linux OS (any supported distro) arch-x64 and removed refs/heads/main labels Dec 28, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tannergooding tannergooding

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI os-linux Linux OS (any supported distro) PGO
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
5 participants
@jeffhandley @jeffschwMSFT @tannergooding @kotlarmilos @JulieLeeMSFT