Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 4 Regressions on 12/15/2022 8:20:21 PM #79877

Closed
performanceautofiler bot opened this issue Dec 20, 2022 · 4 comments
Closed

[Perf] Linux/arm64: 4 Regressions on 12/15/2022 8:20:21 PM #79877

performanceautofiler bot opened this issue Dec 20, 2022 · 4 comments
Assignees
@vargaz
Labels
arch-arm64 area-Codegen-Interpreter-mono
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline ef903fd300fc5115360fc27fc41207f232a5c87d
Compare c6045adbf8f9092acc8c273946a3398f6309b80e
Diff Diff

Regressions in System.Collections.CopyTo<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 2.10 μs 2.27 μs 1.08 0.09 False
ReadOnlyMemory - Duration of single invocation 2.27 μs 2.42 μs 1.06 0.09 False
ReadOnlySpan - Duration of single invocation 2.09 μs 2.27 μs 1.09 0.09 False
Memory - Duration of single invocation 2.25 μs 2.42 μs 1.08 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<String>.Span(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -52.11637039562526 (T) = (0 -2264.434240995673) / Math.Sqrt((46.39408208861247 / (7)) + (11.57166468987502 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.08052533386203431 = (2095.678990609215 - 2264.434240995673) / 2095.678990609215 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ReadOnlyMemory(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -38.51805016110958 (T) = (0 -2416.5377627207945) / Math.Sqrt((127.11716455370656 / (7)) + (2.2028853276947356 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.0744413266866325 = (2249.110959062721 - 2416.5377627207945) / 2249.110959062721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ReadOnlySpan(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -37.25621453581531 (T) = (0 -2261.3970837957963) / Math.Sqrt((70.3671051245492 / (7)) + (30.427218130467224 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.07995544572223148 = (2093.972573362472 - 2261.3970837957963) / 2093.972573362472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.Memory(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -56.79796911005948 (T) = (0 -2421.874213720123) / Math.Sqrt((59.021261496598896 / (7)) + (1.9411048527997075 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.07603585876375796 = (2250.7374582317166 - 2421.874213720123) / 2250.7374582317166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added arm64 untriaged New issue has not been triaged by the area owner labels Dec 20, 2022
@kotlarmilos
Copy link
Member

Probably 6fecc25

@kotlarmilos kotlarmilos transferred this issue from dotnet/perf-autofiling-issues Dec 21, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@kotlarmilos kotlarmilos added area-Codegen-Interpreter-mono and removed untriaged New issue has not been triaged by the area owner labels Dec 21, 2022
@ghost
Copy link

ghost commented Dec 21, 2022

Tagging subscribers to this area: @BrzVlad
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline ef903fd300fc5115360fc27fc41207f232a5c87d
Compare c6045adbf8f9092acc8c273946a3398f6309b80e
Diff Diff

Regressions in System.Collections.CopyTo<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Span - Duration of single invocation 2.10 μs 2.27 μs 1.08 0.09 False
ReadOnlyMemory - Duration of single invocation 2.27 μs 2.42 μs 1.06 0.09 False
ReadOnlySpan - Duration of single invocation 2.09 μs 2.27 μs 1.09 0.09 False
Memory - Duration of single invocation 2.25 μs 2.42 μs 1.08 0.09 False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CopyTo&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CopyTo<String>.Span(Size: 2048)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -52.11637039562526 (T) = (0 -2264.434240995673) / Math.Sqrt((46.39408208861247 / (7)) + (11.57166468987502 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.08052533386203431 = (2095.678990609215 - 2264.434240995673) / 2095.678990609215 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ReadOnlyMemory(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -38.51805016110958 (T) = (0 -2416.5377627207945) / Math.Sqrt((127.11716455370656 / (7)) + (2.2028853276947356 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.0744413266866325 = (2249.110959062721 - 2416.5377627207945) / 2249.110959062721 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.ReadOnlySpan(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -37.25621453581531 (T) = (0 -2261.3970837957963) / Math.Sqrt((70.3671051245492 / (7)) + (30.427218130467224 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.07995544572223148 = (2093.972573362472 - 2261.3970837957963) / 2093.972573362472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Collections.CopyTo&lt;String&gt;.Memory(Size: 2048)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 11/17/2022 1:45:21 AM, 12/15/2022 1:59:35 AM, 12/19/2022 10:34:07 PM falls between 12/10/2022 11:25:25 PM and 12/19/2022 10:34:07 PM.
IsRegressionStdDev: Marked as regression because -56.79796911005948 (T) = (0 -2421.874213720123) / Math.Sqrt((59.021261496598896 / (7)) + (1.9411048527997075 / (3))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (7) + (3) - 2, .025) and -0.07603585876375796 = (2250.7374582317166 - 2421.874213720123) / 2250.7374582317166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: vargaz
Labels:

untriaged, area-Codegen-Interpreter-mono, refs/heads/main, Regression, arm64
Milestone: -

@kotlarmilos kotlarmilos added this to the 8.0.0 milestone Dec 21, 2022
@kotlarmilos kotlarmilos mentioned this issue Dec 21, 2022
Closed
@kotlarmilos
Copy link
Member

Doesn't look like a regression according to a history trend. Closing the issue.

@ghost ghost locked as resolved and limited conversation to collaborators Jan 27, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vargaz vargaz

Labels
arch-arm64 area-Codegen-Interpreter-mono
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests
3 participants
@vargaz @jeffhandley @kotlarmilos