Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in System.Globalization.Tests.StringSearch #66344

Closed
performanceautofiler bot opened this issue Mar 8, 2022 · 5 comments
Closed

Regressions in System.Globalization.Tests.StringSearch #66344

performanceautofiler bot opened this issue Mar 8, 2022 · 5 comments
Assignees
@EgorBo
Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 8, 2022 •
edited by BruceForstall
Loading

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 093bdc466693e847dc75b6fd86d895bf603acdee
Compare 24ce527fb5f966a8262e48b5812cefc240ba6ef5
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_DifferentFirstChar - Duration of single invocation 16.54 ns 19.27 ns 1.16 0.17 False
IsSuffix_SecondHalf - Duration of single invocation 11.05 ns 13.49 ns 1.22 0.19 False
IsPrefix_DifferentFirstChar - Duration of single invocation 16.62 ns 20.49 ns 1.23 0.14 True
IsPrefix_DifferentFirstChar - Duration of single invocation 7.28 ns 9.31 ns 1.28 0.08 True
IsSuffix_DifferentLastChar - Duration of single invocation 17.33 ns 22.19 ns 1.28 0.10 True

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.266005117342086 > 17.319421944670815.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -16.17775162565399 (T) = (0 -19.439132430468234) / Math.Sqrt((0.33840451347766215 / (32)) + (0.8196781723279472 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.19064243157424285 = (16.32659135519487 - 19.439132430468234) / 16.32659135519487 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ordinal, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.494035145405856 > 11.423817835667046.
IsChangePoint: Marked as a change because one of 2/7/2022 2:49:00 PM, 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -10.843217356825154 (T) = (0 -13.930713108190307) / Math.Sqrt((0.2687027837738012 / (32)) + (1.8707766909290975 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.2564103905936014 = (11.087709248893292 - 13.930713108190307) / 11.087709248893292 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.492591463854897 > 17.46981615354641.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -22.938825723341076 (T) = (0 -20.395362994630986) / Math.Sqrt((0.18082107920220317 / (32)) + (0.5844636704709993 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.21368085957007976 = (16.804551899957954 - 20.395362994630986) / 16.804551899957954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.306066728951546 > 7.630648307594432.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -35.96149304365351 (T) = (0 -9.187458913074286) / Math.Sqrt((0.02836074177823468 / (32)) + (0.053501627579287475 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.25007907988774414 = (7.3495021722140255 - 9.187458913074286) / 7.3495021722140255 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.187752259436763 > 17.8672668603724.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -22.053765728368877 (T) = (0 -20.73112526693323) / Math.Sqrt((0.2062830245172223 / (32)) + (0.6424846978814125 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.21264289256274935 = (17.09582053717474 - 20.73112526693323) / 17.09582053717474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

category:performance
theme:benchmarks
skill-level:intermediate
cost:small
impact:small
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Mar 8, 2022
@kunalspathak kunalspathak changed the title [Perf] Changes at 3/2/2022 5:56:50 PM Regressions in System.Globalization.Tests.StringSearch Mar 8, 2022
@dotnet-issue-labeler dotnet-issue-labeler bot added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Mar 8, 2022
@kunalspathak kunalspathak transferred this issue from dotnet/perf-autofiling-issues Mar 8, 2022
@ghost
Copy link

ghost commented Mar 8, 2022

Tagging subscribers to this area: @JulieLeeMSFT
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS Windows 10.0.18362
Baseline 093bdc466693e847dc75b6fd86d895bf603acdee
Compare 24ce527fb5f966a8262e48b5812cefc240ba6ef5
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsPrefix_DifferentFirstChar - Duration of single invocation 16.54 ns 19.27 ns 1.16 0.17 False
IsSuffix_SecondHalf - Duration of single invocation 11.05 ns 13.49 ns 1.22 0.19 False
IsPrefix_DifferentFirstChar - Duration of single invocation 16.62 ns 20.49 ns 1.23 0.14 True
IsPrefix_DifferentFirstChar - Duration of single invocation 7.28 ns 9.31 ns 1.28 0.08 True
IsSuffix_DifferentLastChar - Duration of single invocation 17.33 ns 22.19 ns 1.28 0.10 True

graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, False))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.266005117342086 > 17.319421944670815.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -16.17775162565399 (T) = (0 -19.439132430468234) / Math.Sqrt((0.33840451347766215 / (32)) + (0.8196781723279472 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.19064243157424285 = (16.32659135519487 - 19.439132430468234) / 16.32659135519487 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, Ordinal, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.494035145405856 > 11.423817835667046.
IsChangePoint: Marked as a change because one of 2/7/2022 2:49:00 PM, 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -10.843217356825154 (T) = (0 -13.930713108190307) / Math.Sqrt((0.2687027837738012 / (32)) + (1.8707766909290975 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.2564103905936014 = (11.087709248893292 - 13.930713108190307) / 11.087709248893292 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 20.492591463854897 > 17.46981615354641.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -22.938825723341076 (T) = (0 -20.395362994630986) / Math.Sqrt((0.18082107920220317 / (32)) + (0.5844636704709993 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.21368085957007976 = (16.804551899957954 - 20.395362994630986) / 16.804551899957954 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, OrdinalIgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.306066728951546 > 7.630648307594432.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -35.96149304365351 (T) = (0 -9.187458913074286) / Math.Sqrt((0.02836074177823468 / (32)) + (0.053501627579287475 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.25007907988774414 = (7.3495021722140255 - 9.187458913074286) / 7.3495021722140255 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, IgnoreCase, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.187752259436763 > 17.8672668603724.
IsChangePoint: Marked as a change because one of 3/2/2022 3:17:47 PM, 3/8/2022 2:08:40 PM falls between 2/27/2022 3:52:50 PM and 3/8/2022 2:08:40 PM.
IsRegressionStdDev: Marked as regression because -22.053765728368877 (T) = (0 -20.73112526693323) / Math.Sqrt((0.2062830245172223 / (32)) + (0.6424846978814125 / (31))) is less than -1.9996235849941724 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (32) + (31) - 2, .025) and -0.21264289256274935 = (17.09582053717474 - 20.73112526693323) / 17.09582053717474 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked as regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-CodeGen-coreclr, untriaged, refs/heads/main, x64, RunKind=micro, Windows 10.0.18362, Regression, CoreClr
Milestone: -

@kunalspathak
Copy link
Member

Introduced in #65288

@kunalspathak kunalspathak added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Mar 8, 2022
@EgorBo EgorBo self-assigned this Mar 8, 2022
@JulieLeeMSFT JulieLeeMSFT removed the untriaged New issue has not been triaged by the area owner label Mar 10, 2022
@JulieLeeMSFT JulieLeeMSFT added this to the 7.0.0 milestone Mar 10, 2022
@jkotas jkotas added arch-x64 and removed x64 labels Mar 14, 2022
@EgorBo
Copy link
Member

EgorBo commented Mar 15, 2022

Windows: dotnet/perf-autofiling-issues#3985

@EgorBo
Copy link
Member

EgorBo commented Jun 21, 2022 •
edited
Loading

Caused by inliner due to additional [Intrinsic] attributes which boost inlining in certain cases. The actual unrolling is not happening here (validated)

@EgorBo EgorBo modified the milestones: 7.0.0, 8.0.0 Jun 21, 2022
@jeffhandley jeffhandley added os-windows runtime-coreclr specific to the CoreCLR runtime and removed Windows 10.0.18362 labels Dec 28, 2022
@EgorBo
Copy link
Member

EgorBo commented Apr 10, 2023

Not relevant any more:

image

@EgorBo EgorBo closed this as completed Apr 10, 2023
@ghost ghost locked as resolved and limited conversation to collaborators May 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@EgorBo EgorBo

Labels
arch-x64 area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
6 participants
@EgorBo @jeffhandley @jkotas @BruceForstall @kunalspathak @JulieLeeMSFT