Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

[performanceautofiler]

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	801e81ee8a5d934a50d91d5dca9ad62a4c708bff
Compare	5ce2b9f860f7a84c3059650bb67817d59d8f4953
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.30 ms	3.50 ms	1.06	0.00	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM.
IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[EgorBo]

Can be one of #66432, #66339 or #66363 cc @stephentoub

[stephentoub]

Based on the specific pattern, it would almost certainly be the switch to use StartsWith(..., OrdinalIgnoreCase) (#66339), since this pattern would have us doing a comparison against 7 different strings case-insensitive. @EgorBo, is your change to lift the inlining/unrolling limit reflected here?

[EgorBo]

@stephentoub if it's StartsWith we need to wait a couple of days to see if #66534 helped (https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_arm64_Windows%2010.0.19041%2fSystem.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern%3a%20%22(%3fi)Sherlock%7cHolmes%7cWatson%22%2c%20Options%3a%20Compiled).html).

It worth noting that my PR to inline StartsWith handles only up to 16 chars for arm64 where we don't have Vector256. But I have a small patch that extended that up to 64 chars for hot blocks (will file a PR this weekend)

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-json
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	801e81ee8a5d934a50d91d5dca9ad62a4c708bff
Compare	5ce2b9f860f7a84c3059650bb67817d59d8f4953
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.30 ms	3.50 ms	1.06	0.00	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM.
IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.Json, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.19041, Regression, CoreClr, arm64
Milestone:	-

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	arm64
OS	Windows 10.0.19041
Baseline	801e81ee8a5d934a50d91d5dca9ad62a4c708bff
Compare	5ce2b9f860f7a84c3059650bb67817d59d8f4953
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.30 ms	3.50 ms	1.06	0.00	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "(?i)Sherlock|Holmes|Watson|Irene|Adler|John|Baker", Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 3.5030039285714287 > 3.4672446442307687.
IsChangePoint: Marked as a change because one of 1/27/2022 1:01:32 PM, 2/8/2022 5:20:12 PM, 3/10/2022 4:49:34 PM, 3/17/2022 12:08:34 AM falls between 3/8/2022 5:51:58 AM and 3/17/2022 12:08:34 AM.
IsRegressionStdDev: Marked as regression because -149.07005023991758 (T) = (0 -3505062.4336392777) / Math.Sqrt((19718548.140993573 / (20)) + (19401302.82784744 / (22))) is less than -2.0210753903043583 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (20) + (22) - 2, .025) and -0.06171162112192499 = (3301331.8907967033 - 3505062.4336392777) / 3301331.8907967033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.RegularExpressions, untriaged, refs/heads/main, RunKind=micro, Windows 10.0.19041, Regression, CoreClr, arm64
Milestone:	-

[am11]

arm64 is a redundant label, it can be deleted (I've updated the two issues it was assigned to).

[danmoseley]

I deleted the label. These things pop up when issues are ported to this repo I believe.

[stephentoub]

The regression is gone.