Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/x64: 15 Regressions on 1/17/2023 8:36:05 PM #12056

Closed
performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Closed

[Perf] Linux/x64: 15 Regressions on 1/17/2023 8:36:05 PM #12056

performanceautofiler bot opened this issue Jan 24, 2023 · 2 comments
Assignees
@vargaz
Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in PerfLabTests.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
FooObjIsNull - Duration of single invocation 87.06 μs 127.69 μs 1.47 0.04 False
ObjFooIsObj - Duration of single invocation 87.06 μs 108.82 μs 1.25 0.08 False
CheckArrayIsInterfaceNo - Duration of single invocation 176.78 μs 195.83 μs 1.11 0.01 False
ObjObjrefValueType - Duration of single invocation 76.51 μs 86.41 μs 1.13 0.03 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf.FooObjIsNull

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 127.69448413908248 > 91.40888507465277.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 11:25:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -28.9407035383385 (T) = (0 -109913.23519190036) / Math.Sqrt((5805.7497845370635 / (23)) + (24921590.834909115 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.2624290567969453 = (87064.8806759676 - 109913.23519190036) / 87064.8806759676 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.ObjFooIsObj

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 108.81936591641865 > 91.40839611979166.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 11:25:18 AM, 11/16/2022 9:45:14 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -40.496587174942064 (T) = (0 -109492.45829234409) / Math.Sqrt((96731.59837774561 / (23)) + (12024841.655870236 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.25660086781854013 = (87133.83946839307 - 109492.45829234409) / 87133.83946839307 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.CheckArrayIsInterfaceNo

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 195.83254263392857 > 184.22278712371258.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 11/8/2022 11:25:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -100.83137339459473 (T) = (0 -195879.24980116045) / Math.Sqrt((869644.4158878047 / (23)) + (124096.48723797388 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.11622210560246657 = (175484.11630446714 - 195879.24980116045) / 175484.11630446714 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf.ObjObjrefValueType

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 86.40626406249999 > 80.33844987042684.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -95.31110440261554 (T) = (0 -86232.43356334075) / Math.Sqrt((691.1470667858915 / (23)) + (414376.7394056013 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.12696366013524882 = (76517.49263413857 - 86232.43356334075) / 76517.49263413857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

### Run Information
Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in PerfLabTests.CastingPerf2.CastingPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IFooFooIsIFoo - Duration of single invocation 173.98 μs 217.57 μs 1.25 0.06 False
FooObjIsNull - Duration of single invocation 174.08 μs 217.53 μs 1.25 0.05 False
ObjrefValueTypeObj - Duration of single invocation 509.07 μs 565.60 μs 1.11 0.15 False
ObjFooIsObj - Duration of single invocation 174.09 μs 217.43 μs 1.25 0.05 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.CastingPerf2.CastingPerf*'

Payloads

Baseline
Compare

Histogram

PerfLabTests.CastingPerf2.CastingPerf.IFooFooIsIFoo

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 217.56564216079056 > 182.78622702083334.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -11.219623666455705 (T) = (0 -218231.23474060287) / Math.Sqrt((290770550.6684509 / (23)) + (18024564.135019735 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.22854318300625162 = (177634.1587005431 - 218231.23474060287) / 177634.1587005431 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.FooObjIsNull

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 217.52687406517094 > 182.75239126302085.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 1/17/2023 2:19:41 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -14.301058548548626 (T) = (0 -217558.42944221353) / Math.Sqrt((176520262.73332503 / (22)) + (23664.512990702326 / (41))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (41) - 2, .025) and -0.22881230975749328 = (177047.7295146472 - 217558.42944221353) / 177047.7295146472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.ObjrefValueTypeObj

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 565.6049194775131 > 535.730699890457.
IsChangePoint: Marked as a change because one of 11/8/2022 6:56:37 AM, 12/15/2022 2:06:03 PM, 1/17/2023 2:19:41 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -8.427803947988366 (T) = (0 -569609.2232507404) / Math.Sqrt((765413277.7654374 / (22)) + (187289202.95704886 / (41))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (41) - 2, .025) and -0.10232248931371353 = (516735.5549512276 - 569609.2232507404) / 516735.5549512276 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### PerfLabTests.CastingPerf2.CastingPerf.ObjFooIsObj

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 217.42712494212964 > 182.75751580381944.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 5:00:10 PM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -34.659874641799696 (T) = (0 -219566.56159647577) / Math.Sqrt((19148.107565657367 / (23)) + (68854374.54365024 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.2612852346637392 = (174081.60784107854 - 219566.56159647577) / 174081.60784107854 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in Benchstone.BenchI.CSieve

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 6.69 ms 7.14 ms 1.07 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.CSieve*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.CSieve.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 7.1367223652777785 > 7.024321093601762.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/15/2022 2:06:03 PM, 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -58.03313082440809 (T) = (0 -7114798.841500496) / Math.Sqrt((169367237.88951254 / (23)) + (1819505654.4093769 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.06303616812370952 = (6692903.830410895 - 7114798.841500496) / 6692903.830410895 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in Span.IndexerBench

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Ref - Duration of single invocation 686.52 ns 816.45 ns 1.19 0.01 False
WriteViaIndexer2 - Duration of single invocation 1.35 μs 1.58 μs 1.16 0.01 False
Fixed2 - Duration of single invocation 684.24 ns 755.88 ns 1.10 0.01 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Span.IndexerBench*'

Payloads

Baseline
Compare

Histogram

Span.IndexerBench.Ref(length: 1024)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 816.4528815823355 > 720.5628896681205.
IsChangePoint: Marked as a change because one of 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -54.01578571233896 (T) = (0 -805.1505202164524) / Math.Sqrt((0.09670833656463537 / (23)) + (193.1690633029895 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.17301088924931715 = (686.3964585458524 - 805.1505202164524) / 686.3964585458524 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.WriteViaIndexer2(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.5762541745082346 > 1.4216933608579483.
IsChangePoint: Marked as a change because one of 11/8/2022 11:25:18 AM, 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -86.57434505755303 (T) = (0 -1577.7515103879755) / Math.Sqrt((117.77631298100852 / (23)) + (57.154930630670115 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.16337130013406476 = (1356.1891291337154 - 1577.7515103879755) / 1356.1891291337154 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### Span.IndexerBench.Fixed2(length: 1024)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 755.8797380212143 > 718.581490615968.
IsChangePoint: Marked as a change because one of 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -43.40132918727742 (T) = (0 -772.0249528251722) / Math.Sqrt((3.4328291836319282 / (23)) + (155.89127421717015 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.12750652938865473 = (684.7188310685632 - 772.0249528251722) / 684.7188310685632 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in Benchstone.BenchI.Fib

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 181.09 μs 192.49 μs 1.06 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.Fib*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchI.Fib.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 192.4872076437282 > 190.0861884594662.
IsChangePoint: Marked as a change because one of 11/2/2022 9:35:40 AM, 12/15/2022 2:06:03 PM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -54.318777521357966 (T) = (0 -192868.74533593334) / Math.Sqrt((440376.28892462613 / (23)) + (1078359.4597404494 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.06436604713838084 = (181205.27787829557 - 192868.74533593334) / 181205.27787829557 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in Benchstone.BenchF.NewtR

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 417.70 ms 455.48 ms 1.09 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchF.NewtR*'

Payloads

Baseline
Compare

Histogram

Benchstone.BenchF.NewtR.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 455.4765384166667 > 438.520834455.
IsChangePoint: Marked as a change because one of 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -38.64762540272537 (T) = (0 -459460126.7308013) / Math.Sqrt((13218900456.46261 / (23)) + (46882578290075.055 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.10021551295080632 = (417609205.94413126 - 459460126.7308013) / 417609205.94413126 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture x64
OS ubuntu 18.04
Baseline e9b9489f41cface8f537f87ab1e187c9244c5ea9
Compare 7b7da67a5892e22f4de9e8ab084b6def17299c8c
Diff Diff

Regressions in Benchstone.MDBenchI.MDLogicArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Test - Duration of single invocation 554.89 ms 587.50 ms 1.06 0.02 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.MDBenchI.MDLogicArray*'

Payloads

Baseline
Compare

Histogram

Benchstone.MDBenchI.MDLogicArray.Test

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 587.5047872307692 > 582.5891596908654.
IsChangePoint: Marked as a change because one of 12/20/2022 9:29:18 AM, 1/17/2023 5:52:57 PM, 1/24/2023 6:33:57 AM falls between 1/15/2023 4:08:29 PM and 1/24/2023 6:33:57 AM.
IsRegressionStdDev: Marked as regression because -213.38336390776198 (T) = (0 -587863674.0174588) / Math.Sqrt((311600832058.2873 / (23)) + (409888704867.0719 / (40))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (23) + (40) - 2, .025) and -0.05931327886517433 = (554947894.7788023 - 587863674.0174588) / 554947894.7788023 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@naricc
Copy link

naricc commented Jan 31, 2023

@vargaz It looks like this caused soem regressions: dotnet/runtime@bd3e1f5

@naricc naricc removed the untriaged label Jan 31, 2023
@naricc naricc mentioned this issue Jan 31, 2023
Closed
@naricc naricc added the LinkedFromMaster Is this issue linked from the master perf issue label Jan 31, 2023
@kotlarmilos
Copy link
Member

Probably mitigated by dotnet/runtime@ce66d01

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vargaz vargaz

Labels
arch-x64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-aot os-linux perf-regression runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vargaz @kotlarmilos @naricc