Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 26 Regressions on 2/2/2023 8:18:56 AM #12546

Closed
performanceautofiler bot opened this issue Feb 7, 2023 · 5 comments
Closed

[Perf] Linux/x64: 26 Regressions on 2/2/2023 8:18:56 AM #12546

performanceautofiler bot opened this issue Feb 7, 2023 · 5 comments
Assignees
@tannergooding @matouskozak
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Feb 7, 2023 •
edited
Loading

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2ba2396495c22429035d165e478672c442f81e22
Compare 6aa9f8b5a5d7ea4d79715f0b16f2a5b0ab6ac48d
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector4

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformVector2ByMatrix4x4Benchmark - Duration of single invocation 61.04 ns 103.67 ns 1.70 0.02 False
TransformVector3ByMatrix4x4Benchmark - Duration of single invocation 62.67 ns 131.29 ns 2.10 0.02 False
TransformByMatrix4x4Benchmark - Duration of single invocation 62.54 ns 131.10 ns 2.10 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector4*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector4.TransformVector2ByMatrix4x4Benchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.6672756414579 > 64.07873414548665.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -261.5121698952436 (T) = (0 -103.70841079210498) / Math.Sqrt((0.8266898976539371 / (32)) + (0.0060794182356499514 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.686704279880091 = (61.48582892045409 - 103.70841079210498) / 61.48582892045409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformVector3ByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 131.29033100885084 > 65.79947882874343.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -1172.2932193497213 (T) = (0 -131.52804097883265) / Math.Sqrt((0.05731244522682109 / (32)) + (0.042595775959747666 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -1.09179228152111 = (62.87815580005299 - 131.52804097883265) / 62.87815580005299 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector4.TransformByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 131.09665621623762 > 65.67273861527045.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -1180.3830147076478 (T) = (0 -131.28491517031983) / Math.Sqrt((0.059688820212212904 / (32)) + (0.03900243675299941 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -1.0902323068787791 = (62.80876758926374 - 131.28491517031983) / 62.80876758926374 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline 2ba2396495c22429035d165e478672c442f81e22
Compare 6aa9f8b5a5d7ea4d79715f0b16f2a5b0ab6ac48d
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformNormalByMatrix4x4Benchmark - Duration of single invocation 57.11 ns 103.26 ns 1.81 0.02 False
ReflectBenchmark - Duration of single invocation 17.85 ns 22.25 ns 1.25 0.03 False
TransformByMatrix3x2Benchmark - Duration of single invocation 20.86 ns 35.02 ns 1.68 0.01 False
TransformNormalByMatrix3x2Benchmark - Duration of single invocation 21.06 ns 30.80 ns 1.46 0.02 False
TransformByMatrix4x4Benchmark - Duration of single invocation 56.64 ns 112.73 ns 1.99 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector2.TransformNormalByMatrix4x4Benchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 103.25827161571465 > 59.38323164289799.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -929.5349636018972 (T) = (0 -103.33505678102675) / Math.Sqrt((0.03932878754621575 / (32)) + (0.03407937097324634 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.8292597641357958 = (56.49009441250446 - 103.33505678102675) / 56.49009441250446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector2.ReflectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.249122924577076 > 18.73320710923984.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -84.91226968529888 (T) = (0 -22.32699740415812) / Math.Sqrt((0.013408309299799852 / (32)) + (0.06130454198198223 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.2506402571183573 = (17.85245379482867 - 22.32699740415812) / 17.85245379482867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector2.TransformByMatrix3x2Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.01741331573861 > 21.924542666276228.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -351.1330893069049 (T) = (0 -35.00126778053213) / Math.Sqrt((0.022328937915959142 / (32)) + (0.023997215196553432 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.6774976696200085 = (20.865166261876666 - 35.00126778053213) / 20.865166261876666 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector2.TransformNormalByMatrix3x2Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 30.80414912919404 > 22.104718261766987.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -1558.7363160637874 (T) = (0 -30.843162676463766) / Math.Sqrt((0.0002750842344857577 / (32)) + (0.0008069837749436126 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.46662497960469346 = (21.030026833974336 - 30.843162676463766) / 21.030026833974336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector2.TransformByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 112.72664567131525 > 59.38311874475532.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -851.0124657548656 (T) = (0 -112.29919020753196) / Math.Sqrt((0.08040684302556385 / (32)) + (0.04623554584157946 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.9857344316007949 = (56.552975272228245 - 112.29919020753196) / 56.552975272228245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2ba2396495c22429035d165e478672c442f81e22
Compare 6aa9f8b5a5d7ea4d79715f0b16f2a5b0ab6ac48d
Diff Diff

Regressions in System.Numerics.Tests.Perf_Quaternion

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
MultiplyByScalarBenchmark - Duration of single invocation 19.07 ns 22.66 ns 1.19 0.01 False
MultiplyByScalarOperatorBenchmark - Duration of single invocation 19.25 ns 20.27 ns 1.05 0.01 False
ConjugateBenchmark - Duration of single invocation 18.17 ns 29.83 ns 1.64 0.03 False
AddOperatorBenchmark - Duration of single invocation 24.26 ns 29.83 ns 1.23 0.01 False
SubtractBenchmark - Duration of single invocation 24.27 ns 29.27 ns 1.21 0.02 False
NegationOperatorBenchmark - Duration of single invocation 18.32 ns 20.94 ns 1.14 0.00 False
SubtractionOperatorBenchmark - Duration of single invocation 24.18 ns 28.42 ns 1.18 0.01 False
NegateBenchmark - Duration of single invocation 18.10 ns 20.99 ns 1.16 0.01 False
InverseBenchmark - Duration of single invocation 26.63 ns 45.15 ns 1.70 0.01 False
LengthBenchmark - Duration of single invocation 13.56 ns 17.00 ns 1.25 0.23 False
LengthSquaredBenchmark - Duration of single invocation 7.12 ns 10.46 ns 1.47 0.14 False
DivideBenchmark - Duration of single invocation 24.18 ns 28.96 ns 1.20 0.01 False
NormalizeBenchmark - Duration of single invocation 31.16 ns 34.51 ns 1.11 0.03 False
AddBenchmark - Duration of single invocation 24.30 ns 28.74 ns 1.18 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Quaternion*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Quaternion.MultiplyByScalarBenchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.656520548467245 > 19.865167401787232.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -13.502206816817468 (T) = (0 -20.317879468869656) / Math.Sqrt((0.006434333889523435 / (32)) + (0.25991861714067416 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.07193692931715709 = (18.954360945296013 - 20.317879468869656) / 18.954360945296013 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.MultiplyByScalarOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.272556593256077 > 20.197487144025185.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -15.690758715040502 (T) = (0 -20.35496484198491) / Math.Sqrt((0.007941564908236255 / (32)) + (0.14150649533687176 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.061741386546394536 = (19.17130207026687 - 20.35496484198491) / 19.17130207026687 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.ConjugateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.826447769730947 > 19.1035057967722.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -56.45568327014482 (T) = (0 -30.71525820509964) / Math.Sqrt((0.007923982435490638 / (32)) + (1.2786156310251646 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.6909813012645896 = (18.16416194675212 - 30.71525820509964) / 18.16416194675212 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.AddOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.83060737301017 > 25.459006944092398.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -41.63276956431989 (T) = (0 -29.187064516891237) / Math.Sqrt((0.028909749229475198 / (32)) + (0.3223782582139687 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.19691367235643467 = (24.38527121127202 - 29.187064516891237) / 24.38527121127202 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.SubtractBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 29.26605625074132 > 25.479165486485897.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -39.32179709501654 (T) = (0 -29.029522493175573) / Math.Sqrt((0.0079847337306911 / (32)) + (0.3728485052885755 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.1956193518943578 = (24.27990350539304 - 29.029522493175573) / 24.27990350539304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.NegationOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.943328272891954 > 19.201109410216464.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -53.75887643040247 (T) = (0 -20.961630249288255) / Math.Sqrt((0.016118288904192705 / (32)) + (0.05083918047216744 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.1457077394765 = (18.29579178618982 - 20.961630249288255) / 18.29579178618982 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.SubtractionOperatorBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.416939252188115 > 25.43036863058595.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -34.36048557906005 (T) = (0 -28.882644199512033) / Math.Sqrt((0.01880319486518378 / (32)) + (0.45198961365808815 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.1897459714276338 = (24.276311828864063 - 28.882644199512033) / 24.276311828864063 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.NegateBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.992734342242493 > 19.102122637923525.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -108.00865741026507 (T) = (0 -20.8052071508445) / Math.Sqrt((0.012117113903008559 / (32)) + (0.005021751063499331 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.14173438716313053 = (18.222458204608547 - 20.8052071508445) / 18.222458204608547 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.InverseBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 45.15046971016388 > 27.974686728636435.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -115.61639346744772 (T) = (0 -45.32337226964604) / Math.Sqrt((0.006441403832642145 / (32)) + (0.6708719331928902 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.6988195230069901 = (26.679333299291 - 45.32337226964604) / 26.679333299291 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.LengthBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.99963102142419 > 14.398894490556444.
IsChangePoint: Marked as a change because one of 12/5/2022 8:20:45 AM, 1/20/2023 8:02:54 AM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -29.941603331705835 (T) = (0 -16.01320078818496) / Math.Sqrt((0.09083215675829355 / (32)) + (0.06648645734191612 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.15921494021915178 = (13.813832303746564 - 16.01320078818496) / 13.813832303746564 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.LengthSquaredBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 10.45523238516379 > 7.475309681878603.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -73.73903267192931 (T) = (0 -10.44530222128667) / Math.Sqrt((0.023064147908512363 / (32)) + (0.032044249786097746 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.4534854191785108 = (7.186382528137232 - 10.44530222128667) / 7.186382528137232 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.DivideBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.958491163081856 > 25.429087464251506.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -56.528077507633505 (T) = (0 -29.089030032872998) / Math.Sqrt((0.021871008416868825 / (32)) + (0.16902725855689038 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.19719650685473097 = (24.297623544939636 - 29.089030032872998) / 24.297623544939636 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.NormalizeBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.51245282340774 > 32.33044362291328.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -45.48892601524981 (T) = (0 -34.35933826213871) / Math.Sqrt((0.02586448365413959 / (32)) + (0.12693730213405396 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.11095060497277758 = (30.927872137916193 - 34.35933826213871) / 30.927872137916193 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Quaternion.AddBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 28.743604756054257 > 25.49864557452467.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -44.95065952590395 (T) = (0 -29.150320138826967) / Math.Sqrt((0.03974355606819438 / (32)) + (0.26986874106819164 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.19938001964815705 = (24.304490371098836 - 29.150320138826967) / 24.304490371098836 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2ba2396495c22429035d165e478672c442f81e22
Compare 6aa9f8b5a5d7ea4d79715f0b16f2a5b0ab6ac48d
Diff Diff

Regressions in System.Numerics.Tests.Perf_Vector3

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
TransformNormalByMatrix4x4Benchmark - Duration of single invocation 60.59 ns 133.61 ns 2.21 0.05 False
ReflectBenchmark - Duration of single invocation 39.56 ns 54.38 ns 1.37 0.05 False
TransformByMatrix4x4Benchmark - Duration of single invocation 60.30 ns 140.45 ns 2.33 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector3*'

Payloads

Baseline
Compare

Histogram

System.Numerics.Tests.Perf_Vector3.TransformNormalByMatrix4x4Benchmark

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 133.60730619724092 > 63.4169362960012.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -205.94858292380965 (T) = (0 -131.86187836472968) / Math.Sqrt((1.0031519981283104 / (32)) + (2.252607832689632 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -1.157424776958311 = (61.12003522581112 - 131.86187836472968) / 61.12003522581112 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.ReflectBenchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 54.37676712879695 > 39.26441875903095.
IsChangePoint: Marked as a change because one of 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -109.2541069816183 (T) = (0 -54.36968752729743) / Math.Sqrt((0.6558516005641938 / (32)) + (0.0704714877918115 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.4411531477782904 = (37.726516165970835 - 54.36968752729743) / 37.726516165970835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Numerics.Tests.Perf_Vector3.TransformByMatrix4x4Benchmark

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 140.45491333938295 > 63.4468529696599.
IsChangePoint: Marked as a change because one of 1/6/2023 6:58:01 PM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -276.13835551821217 (T) = (0 -141.28510087200408) / Math.Sqrt((1.1188050406751187 / (32)) + (1.2873126277650586 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -1.3151324398542408 = (61.02678984572446 - 141.28510087200408) / 61.02678984572446 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline 2ba2396495c22429035d165e478672c442f81e22
Compare 6aa9f8b5a5d7ea4d79715f0b16f2a5b0ab6ac48d
Diff Diff

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
BitArraySet - Duration of single invocation 1.18 μs 1.25 μs 1.06 0.00 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Perf_BitArray.BitArraySet(Size: 512)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.2516630014565961 > 1.2444290418657364.
IsChangePoint: Marked as a change because one of 11/21/2022 6:40:27 PM, 1/3/2023 6:29:42 AM, 1/17/2023 5:52:57 PM, 1/20/2023 8:02:54 AM, 2/2/2023 5:46:53 AM, 2/7/2023 8:22:47 AM falls between 1/29/2023 5:12:48 PM and 2/7/2023 8:22:47 AM.
IsRegressionStdDev: Marked as regression because -107.70423255441933 (T) = (0 -1249.7674070202208) / Math.Sqrt((1.170859898871218 / (32)) + (8.31256259579378 / (26))) is less than -2.0032407188469383 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (26) - 2, .025) and -0.05423122105795802 = (1185.4775139044311 - 1249.7674070202208) / 1185.4775139044311 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added LLVM=false PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Feb 7, 2023
@naricc
Copy link

naricc commented Feb 8, 2023

@tannergooding It looks like mono saw some regressions around dotnet/runtime@4354abd...9e8d0a8, but I think this is part of a larger mono issue around simd/vectors? Is that right @fanyang-mono

@naricc naricc removed the untriaged label Feb 8, 2023
@tannergooding
Copy link
Member

Right, and it's something I'm working on fixing/improving: dotnet/runtime#81541

@SamMonoRT
Copy link
Member

Assigning to @tannergooding for accurate tracking

@kotlarmilos kotlarmilos mentioned this issue Feb 24, 2023
Closed
@kotlarmilos kotlarmilos added the LinkedFromMaster Is this issue linked from the master perf issue label Feb 24, 2023
@kotlarmilos kotlarmilos mentioned this issue Feb 24, 2023
Closed
@kotlarmilos kotlarmilos mentioned this issue Apr 4, 2023
Closed
@kotlarmilos
Copy link
Member

@matouskozak Similar to dotnet/runtime#80569.

/cc: @SamMonoRT

@matouskozak
Copy link
Member

These microbenchmarks are no longer regressed in AOT-llvm scenario.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tannergooding tannergooding

@matouskozak matouskozak

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@tannergooding @kotlarmilos @naricc @SamMonoRT @matouskozak