Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Perf] Linux/arm64: 233 Regressions on 3/8/2023 11:14:40 PM #14003

Closed
performanceautofiler bot opened this issue Mar 14, 2023 · 13 comments
Closed

[Perf] Linux/arm64: 233 Regressions on 3/8/2023 11:14:40 PM #14003

performanceautofiler bot opened this issue Mar 14, 2023 · 13 comments
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono

Comments

@performanceautofiler
Copy link

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Segment

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadMultiSegmentSequence - Duration of single invocation 188.54 μs 249.62 μs 1.32 0.02 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 16.52 ms 23.30 ms 1.41 0.01 False
ReadSingleSegmentSequence - Duration of single invocation 16.73 ms 20.99 ms 1.25 0.02 False
ReadMultiSegmentSequence - Duration of single invocation 18.45 ms 24.77 ms 1.34 0.01 False
ReadSingleSegmentSequenceByN - Duration of single invocation 162.30 μs 226.74 μs 1.40 0.03 False
ReadMultiSegmentSequence - Duration of single invocation 163.03 μs 220.09 μs 1.35 0.02 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 167.01 μs 231.41 μs 1.39 0.01 False
ReadSingleSegmentSequenceByN - Duration of single invocation 1.61 ms 2.23 ms 1.38 0.01 False
ReadSingleSegmentSequenceByN - Duration of single invocation 163.04 μs 222.68 μs 1.37 0.02 False
ReadMultiSegmentSequence - Duration of single invocation 18.39 ms 24.61 ms 1.34 0.01 False
ReadMultiSegmentSequence - Duration of single invocation 1.89 ms 2.49 ms 1.32 0.03 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 16.45 ms 23.05 ms 1.40 0.01 False
ReadSingleSegmentSequence - Duration of single invocation 162.54 μs 220.79 μs 1.36 0.02 False
ReadSingleSegmentSequenceByN - Duration of single invocation 16.13 ms 22.43 ms 1.39 0.02 False
ReadMultiSegmentSequence - Duration of single invocation 1.84 ms 2.47 ms 1.34 0.02 False
ReadSingleSegmentSequenceByN - Duration of single invocation 16.20 ms 22.39 ms 1.38 0.02 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 166.99 μs 233.30 μs 1.40 0.02 False
ReadSingleSegmentSequence - Duration of single invocation 1.61 ms 2.15 ms 1.33 0.02 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 1.65 ms 2.34 ms 1.41 0.02 False
ReadSingleSegmentSequenceByN - Duration of single invocation 1.60 ms 2.20 ms 1.38 0.01 False
ReadMultiSegmentSequenceUsingSpan - Duration of single invocation 1.65 ms 2.30 ms 1.39 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Segment*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Segment*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Segment*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Segment*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 4096, TestCase: Json4KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -34.03203547963641 (T) = (0 -246905.0891440524) / Math.Sqrt((12471804.24215335 / (6)) + (5030177.188035249 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.336465215433656 = (184744.8675003013 - 246905.0891440524) / 184744.8675003013 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 4096, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -40.59767791478334 (T) = (0 -22954094.27186147) / Math.Sqrt((76618921368.8188 / (6)) + (55008726066.32455 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4045620122364441 = (16342528.184506657 - 22954094.27186147) / 16342528.184506657 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequence(TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.877803096379836 (T) = (0 -20724857.70684975) / Math.Sqrt((65528158845.908066 / (6)) + (111643098036.28888 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2780036395107869 = (16216587.391553218 - 20724857.70684975) / 16216587.391553218 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 4096, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -30.115308787503654 (T) = (0 -24467747.635) / Math.Sqrt((174544134778.77255 / (6)) + (47022805748.120125 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.33112199558535205 = (18381296.166802857 - 24467747.635) / 18381296.166802857 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 4096, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -14.274649569990125 (T) = (0 -216046.00672629644) / Math.Sqrt((7803330.360065043 / (6)) + (54616861.03627646 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.34320264270165557 = (160843.94108379024 - 216046.00672629644) / 160843.94108379024 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 8192, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -13.988210707623182 (T) = (0 -210318.27286647516) / Math.Sqrt((7409333.967719892 / (6)) + (43948913.64308743 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3029637955033386 = (161415.2853611935 - 210318.27286647516) / 161415.2853611935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 4096, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -44.674358597285156 (T) = (0 -228656.54912034445) / Math.Sqrt((5844820.40611202 / (6)) + (3708527.883893238 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.36873667502427787 = (167056.63937607917 - 228656.54912034445) / 167056.63937607917 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.000568228091886 (T) = (0 -2172142.423940826) / Math.Sqrt((640806595.6203196 / (6)) + (1545860225.4088085 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3621177951412228 = (1594680.3071577381 - 2172142.423940826) / 1594680.3071577381 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.101990914345297 (T) = (0 -214302.3359365397) / Math.Sqrt((9698940.671786878 / (6)) + (38472038.30053333 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3366174929865235 = (160331.83544359045 - 214302.3359365397) / 160331.83544359045 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 8192, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -33.01881394080202 (T) = (0 -24322600.537362643) / Math.Sqrt((113319886228.48116 / (6)) + (61006748299.418335 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.33480347495895035 = (18221858.868108388 - 24322600.537362643) / 18221858.868108388 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 8192, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.586283266699247 (T) = (0 -2434071.6576849488) / Math.Sqrt((2535197937.428427 / (6)) + (1686969385.978403 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32581349773388313 = (1835908.038231117 - 2434071.6576849488) / 1835908.038231117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 8192, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -42.21647800801341 (T) = (0 -22718686.780927405) / Math.Sqrt((66360094544.05675 / (6)) + (48833328207.39257 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3955824900730659 = (16278999.588005697 - 22718686.780927405) / 16278999.588005697 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequence(TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.544000385051014 (T) = (0 -213566.75988120167) / Math.Sqrt((6552653.123245173 / (6)) + (23974459.339909326 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3220465561133259 = (161542.54091403927 - 213566.75988120167) / 161542.54091403927 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 8192, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.506361612836056 (T) = (0 -21713544.47152015) / Math.Sqrt((58598895645.40456 / (6)) + (272746633608.85977 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3581023816475687 = (15988149.910449736 - 21713544.47152015) / 15988149.910449736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequence(segmentSize: 4096, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.472388854113003 (T) = (0 -2427403.0674479166) / Math.Sqrt((1015120824.486639 / (6)) + (1380721357.8507023 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32861461258850877 = (1827018.1920690034 - 2427403.0674479166) / 1827018.1920690034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 4096, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.71306347895022 (T) = (0 -21844100.393015318) / Math.Sqrt((57279095953.529434 / (6)) + (185705596736.83676 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3654835752789697 = (15997336.612820514 - 21844100.393015318) / 15997336.612820514 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 8192, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -32.92625837969159 (T) = (0 -229459.89311055833) / Math.Sqrt((4447927.090367439 / (6)) + (12240316.919642575 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3884531328832806 = (165262.97335946717 - 229459.89311055833) / 165262.97335946717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequence(TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -14.399351599709895 (T) = (0 -2065402.955106027) / Math.Sqrt((728421502.3991449 / (6)) + (3529733204.9974227 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2835104957168485 = (1609182.7546392495 - 2065402.955106027) / 1609182.7546392495 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 8192, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.59183497276481 (T) = (0 -2267507.9128640518) / Math.Sqrt((882845895.7742357 / (6)) + (2123436145.0027502 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3935242604182007 = (1627175.0534027775 - 2267507.9128640518) / 1627175.0534027775 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadSingleSegmentSequenceByN(numberOfBytes: 4096, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -36.84738359600007 (T) = (0 -2174908.979082961) / Math.Sqrt((857381443.2889665 / (6)) + (409504644.9141906 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.36116003986646233 = (1597834.8727430555 - 2174908.979082961) / 1597834.8727430555 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Segment.ReadMultiSegmentSequenceUsingSpan(segmentSize: 4096, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -43.912164725010534 (T) = (0 -2272889.955027636) / Math.Sqrt((527753140.17146033 / (6)) + (458151734.38822347 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.37916920848068675 = (1648013.8485193455 - 2272889.955027636) / 1648013.8485193455 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 24.98 μs 27.66 μs 1.11 0.02 False
SystemTextJson_Reflection_ - Duration of single invocation 24.95 μs 27.42 μs 1.10 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<Location>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -27.79341350322984 (T) = (0 -27457.546564516982) / Math.Sqrt((19430.954000250997 / (6)) + (21675.816903124676 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1039764218057658 = (24871.497273107412 - 27457.546564516982) / 24871.497273107412 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;Location&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.769956795187426 (T) = (0 -27480.448689789373) / Math.Sqrt((28082.25899418412 / (6)) + (36766.4086278914 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1081443761940573 = (24798.61765320823 - 27480.448689789373) / 24798.61765320823 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 8.90 μs 10.30 μs 1.16 0.03 False
SystemTextJson_SourceGen_ - Duration of single invocation 8.91 μs 10.48 μs 1.18 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<LoginViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.22817480164954 (T) = (0 -10330.650887316264) / Math.Sqrt((4486.702362910643 / (6)) + (23251.51116423504 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.16675057444501917 = (8854.206814708601 - 10330.650887316264) / 8854.206814708601 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;LoginViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.887445453826329 (T) = (0 -10287.713815326973) / Math.Sqrt((20328.69696551852 / (6)) + (16612.59190655359 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15487950357520966 = (8908.040867881766 - 10287.713815326973) / 8908.040867881766 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 556.66 μs 848.13 μs 1.52 0.04 False
SystemTextJson_Reflection_ - Duration of single invocation 553.46 μs 841.91 μs 1.52 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<IndexViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -65.10606739246313 (T) = (0 -840487.7757289859) / Math.Sqrt((72973583.74850419 / (6)) + (26189806.55934347 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.503896956895454 = (558873.2471831272 - 840487.7757289859) / 558873.2471831272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;IndexViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -68.42918044640373 (T) = (0 -840038.3200306455) / Math.Sqrt((7756883.556311434 / (6)) + (65277098.80972254 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.5194324352500115 = (552863.2274408591 - 840038.3200306455) / 552863.2274408591 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Baseline - Duration of single invocation 24.92 μs 27.38 μs 1.10 0.02 False
MissingProperties - Duration of single invocation 17.14 μs 19.72 μs 1.15 0.03 False
CaseInsensitiveNotMatching - Duration of single invocation 24.91 μs 27.56 μs 1.11 0.03 False
CaseInsensitiveMatching - Duration of single invocation 24.86 μs 27.84 μs 1.12 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive<Location>.Baseline

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -35.862173711892865 (T) = (0 -27319.906207330132) / Math.Sqrt((20413.164585907347 / (6)) + (5727.877469049857 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1004343339562584 = (24826.475659942542 - 27319.906207330132) / 24826.475659942542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;.MissingProperties

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -9.994445930897932 (T) = (0 -19986.582213548085) / Math.Sqrt((193575.8049516094 / (6)) + (150162.4021447095 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15222874547464163 = (17346.019435849903 - 19986.582213548085) / 17346.019435849903 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;.CaseInsensitiveNotMatching

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -34.98302275557726 (T) = (0 -27479.132286898817) / Math.Sqrt((18341.498964507224 / (6)) + (10738.750613181795 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10676430714873032 = (24828.350633832004 - 27479.132286898817) / 24828.350633832004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadMissingAndCaseInsensitive&lt;Location&gt;.CaseInsensitiveMatching

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.52490893610495 (T) = (0 -27758.69187906636) / Math.Sqrt((13212.87429619894 / (6)) + (207809.97861207082 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11731967919776783 = (24844.001583322166 - 27758.69187906636) / 24844.001583322166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_DocumentParse

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 24.93 ms 30.35 ms 1.22 0.03 False
Parse - Duration of single invocation 73.36 μs 80.71 μs 1.10 0.07 False
Parse - Duration of single invocation 4.99 μs 5.40 μs 1.08 0.04 False
Parse - Duration of single invocation 26.82 μs 32.72 μs 1.22 0.03 False
Parse - Duration of single invocation 6.99 μs 7.45 μs 1.07 0.04 False
Parse - Duration of single invocation 19.99 ms 25.43 ms 1.27 0.02 False
Parse - Duration of single invocation 78.53 μs 84.87 μs 1.08 0.03 False
Parse - Duration of single invocation 26.34 μs 28.02 μs 1.06 0.02 False
Parse - Duration of single invocation 31.77 μs 37.47 μs 1.18 0.03 False
Parse - Duration of single invocation 20.88 μs 22.97 μs 1.10 0.03 False
Parse - Duration of single invocation 7.20 μs 7.70 μs 1.07 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_DocumentParse*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/6/2022 2:12:43 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.5048694867082 (T) = (0 -29962560.368211996) / Math.Sqrt((97299573469.16109 / (6)) + (71028698066.76642 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2126393663106859 = (24708549.961865082 - 29962560.368211996) / 24708549.961865082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.462026683799774 (T) = (0 -79963.39312198882) / Math.Sqrt((2535309.3631813736 / (6)) + (295509.6023950595 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.080568250610686 = (74001.24247292783 - 79963.39312198882) / 74001.24247292783 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/4/2022 6:53:17 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.262491135859705 (T) = (0 -5416.237373263077) / Math.Sqrt((770.9888458140864 / (6)) + (983.6957721971829 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08640709215370851 = (4985.458409081123 - 5416.237373263077) / 4985.458409081123 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -43.58468128965672 (T) = (0 -32445.680725568207) / Math.Sqrt((41852.63604755748 / (6)) + (41636.26749356831 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.21523833574314072 = (26699.026661076383 - 32445.680725568207) / 26699.026661076383 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.767264267767056 (T) = (0 -7465.7398154177845) / Math.Sqrt((14486.93773201772 / (6)) + (4387.082481154446 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.05676079758837087 = (7064.739563064145 - 7465.7398154177845) / 7064.739563064145 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.155914315111996 (T) = (0 -25720972.921153847) / Math.Sqrt((38995359285.63539 / (6)) + (256916661471.56998 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2971621080781229 = (19828649.604375258 - 25720972.921153847) / 19828649.604375258 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.233354142268116 (T) = (0 -85148.72253741443) / Math.Sqrt((123588.12805565551 / (6)) + (609753.6673038425 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08613565931136231 = (78396.0289006632 - 85148.72253741443) / 78396.0289006632 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: BasicJson)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 10/7/2022 12:48:42 AM, 12/8/2022 2:00:38 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.139974399226238 (T) = (0 -27858.956163686948) / Math.Sqrt((159563.89303047318 / (6)) + (12116.014337671519 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.07390971705589741 = (25941.618481730227 - 27858.956163686948) / 25941.618481730227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/7/2022 1:52:16 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.874006439104834 (T) = (0 -37586.1131986847) / Math.Sqrt((494629.39671619714 / (6)) + (406280.5315395422 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1722342391717601 = (32063.654125340257 - 37586.1131986847) / 32063.654125340257 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: False, TestRandomAccess: False, TestCase: BasicJson)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.49892330232741 (T) = (0 -23372.163475866513) / Math.Sqrt((40608.55310622523 / (6)) + (343569.1829515874 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1244684645117529 = (20785.076872755846 - 23372.163475866513) / 20785.076872755846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_DocumentParse.Parse(IsDataIndented: True, TestRandomAccess: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/7/2023 1:52:33 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.001408602869866 (T) = (0 -7748.1964952479775) / Math.Sqrt((26101.399919297077 / (5)) + (2064.7798866415824 / (5))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (5) + (5) - 2, .025) and -0.06172235736492405 = (7297.761454772539 - 7748.1964952479775) / 7297.761454772539 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ArrayList>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 306.07 μs 434.87 μs 1.42 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 288.68 μs 394.10 μs 1.37 0.05 False
DeserializeFromReader - Duration of single invocation 351.92 μs 529.42 μs 1.50 0.06 False
DeserializeFromStream - Duration of single invocation 301.85 μs 439.02 μs 1.45 0.06 False
DeserializeFromReader - Duration of single invocation 352.37 μs 549.00 μs 1.56 0.05 False
DeserializeFromUtf8Bytes - Duration of single invocation 288.68 μs 394.76 μs 1.37 0.05 False
DeserializeFromString - Duration of single invocation 305.31 μs 436.30 μs 1.43 0.02 False
DeserializeFromStream - Duration of single invocation 305.19 μs 422.14 μs 1.38 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ArrayList>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -41.486181393074354 (T) = (0 -430859.3549134965) / Math.Sqrt((42417069.79806468 / (6)) + (7278551.819167863 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4026755612935666 = (307169.64549959946 - 430859.3549134965) / 307169.64549959946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.626781792306687 (T) = (0 -386805.3021535132) / Math.Sqrt((60054925.32231599 / (6)) + (31914757.236095477 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3299571757809393 = (290840.4189228006 - 386805.3021535132) / 290840.4189228006 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.806122083934692 (T) = (0 -506671.9822045111) / Math.Sqrt((5052819.395783842 / (6)) + (243941609.5008732 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.44377533460199103 = (350935.4745585722 - 506671.9822045111) / 350935.4745585722 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.295142950764582 (T) = (0 -428616.67900098476) / Math.Sqrt((6409372.026271242 / (6)) + (105924687.58344786 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.42351461090067344 = (301097.4918829911 - 428616.67900098476) / 301097.4918829911 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -13.126716486368686 (T) = (0 -516496.3750581389) / Math.Sqrt((102247797.30898468 / (6)) + (533671824.53645724 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4529480108464025 = (355481.66293800034 - 516496.3750581389) / 355481.66293800034 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.404479900391948 (T) = (0 -384436.0103042888) / Math.Sqrt((2170749.3900249456 / (6)) + (51636565.830399156 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3337063835703389 = (288246.3599485455 - 384436.0103042888) / 288246.3599485455 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -50.732986942023075 (T) = (0 -431146.245624866) / Math.Sqrt((14870891.659971446 / (6)) + (15543933.623290127 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4221851130130584 = (303157.6140685613 - 431146.245624866) / 303157.6140685613 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ArrayList&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -39.19616422447765 (T) = (0 -424208.93706389866) / Math.Sqrt((7276313.333467054 / (6)) + (34401688.82450634 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.40735286874499343 = (301423.2936777163 - 424208.93706389866) / 301423.2936777163 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 5.55 ms 7.08 ms 1.28 0.03 False
SystemTextJson_Reflection_ - Duration of single invocation 5.50 ms 7.01 ms 1.28 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.889928996304132 (T) = (0 -6996697.048416289) / Math.Sqrt((762861350.3670757 / (6)) + (9049765605.8711 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2639509830033372 = (5535576.254540415 - 6996697.048416289) / 5535576.254540415 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -73.65359316979395 (T) = (0 -6993758.726110858) / Math.Sqrt((1758850287.4982712 / (6)) + (540530920.7624241 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.27868036411082636 = (5469512.884069511 - 6993758.726110858) / 5469512.884069511 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Hashtable>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 521.89 μs 722.30 μs 1.38 0.03 False
DeserializeFromStream - Duration of single invocation 510.93 μs 699.47 μs 1.37 0.03 False
DeserializeFromStream - Duration of single invocation 517.76 μs 693.45 μs 1.34 0.07 False
DeserializeFromReader - Duration of single invocation 621.49 μs 876.21 μs 1.41 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 499.70 μs 671.56 μs 1.34 0.04 False
DeserializeFromString - Duration of single invocation 525.38 μs 719.57 μs 1.37 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 498.68 μs 694.64 μs 1.39 0.04 False
DeserializeFromReader - Duration of single invocation 622.61 μs 883.34 μs 1.42 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Hashtable>.DeserializeFromString(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -79.41719311998808 (T) = (0 -722019.5354144553) / Math.Sqrt((35905250.22007639 / (6)) + (688329.164114077 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.37535232384379646 = (524970.6005487926 - 722019.5354144553) / 524970.6005487926 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -37.29067624224964 (T) = (0 -695405.0191107353) / Math.Sqrt((3832638.69416555 / (6)) + (96487453.19844198 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3639500755895114 = (509846.38775006117 - 695405.0191107353) / 509846.38775006117 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -63.850649766535426 (T) = (0 -690074.761254529) / Math.Sqrt((21270270.076142002 / (6)) + (17729994.28341321 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3538004977001677 = (509731.50211336603 - 690074.761254529) / 509731.50211336603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -23.24623858244223 (T) = (0 -887757.296792875) / Math.Sqrt((163479936.507275 / (6)) + (414068244.86711067 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.42741860458801473 = (621931.992436866 - 887757.296792875) / 621931.992436866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.281903735266404 (T) = (0 -658037.5206295289) / Math.Sqrt((17903648.346069843 / (6)) + (139183648.75494188 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.325365955541302 = (496494.9626767614 - 658037.5206295289) / 496494.9626767614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -166.75850483906936 (T) = (0 -719637.7794473756) / Math.Sqrt((3957331.3695294545 / (6)) + (2936078.2188534173 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.37656016436781153 = (522779.7506241733 - 719637.7794473756) / 522779.7506241733 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -14.394199126452422 (T) = (0 -667768.0645748271) / Math.Sqrt((121621148.2457583 / (6)) + (405166180.9174261 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3117546322574785 = (509064.76573718997 - 667768.0645748271) / 509064.76573718997 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Hashtable&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -50.09672818659885 (T) = (0 -879700.7459688719) / Math.Sqrt((33516026.102528572 / (6)) + (87522180.64793418 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4254219546837639 = (617151.1130990243 - 879700.7459688719) / 617151.1130990243 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 356.40 μs 473.16 μs 1.33 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 340.08 μs 451.07 μs 1.33 0.03 False
DeserializeFromString - Duration of single invocation 360.20 μs 503.38 μs 1.40 0.04 False
DeserializeFromReader - Duration of single invocation 455.83 μs 691.86 μs 1.52 0.07 False
DeserializeFromString - Duration of single invocation 361.25 μs 492.43 μs 1.36 0.03 False
DeserializeFromStream - Duration of single invocation 356.16 μs 475.63 μs 1.34 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 332.84 μs 445.68 μs 1.34 0.03 False
DeserializeFromReader - Duration of single invocation 457.51 μs 680.61 μs 1.49 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Dictionary<String, String>>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -55.842552589481556 (T) = (0 -470098.94915991666) / Math.Sqrt((18005769.82455436 / (6)) + (5021986.445234398 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32463604518232675 = (354889.1417153086 - 470098.94915991666) / 354889.1417153086 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.083930441218534 (T) = (0 -440987.85607390245) / Math.Sqrt((11128309.386718417 / (6)) + (49991416.87968304 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.31798496743234955 = (334592.477889197 - 440987.85607390245) / 334592.477889197 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -35.30138679006863 (T) = (0 -492751.6335736372) / Math.Sqrt((8892623.999828164 / (6)) + (50454368.647093035 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3679365275924918 = (360215.27580732014 - 492751.6335736372) / 360215.27580732014 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.992575689315267 (T) = (0 -667849.6857114863) / Math.Sqrt((15115490.52995252 / (6)) + (448256316.70640945 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.471546823606822 = (453841.95392067725 - 667849.6857114863) / 453841.95392067725 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -45.552559470064914 (T) = (0 -491694.9395579067) / Math.Sqrt((42501578.354099914 / (6)) + (2794406.5276917927 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.34851769546800676 = (364618.82644206804 - 491694.9395579067) / 364618.82644206804 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -38.54063621794815 (T) = (0 -469319.25807883526) / Math.Sqrt((24719779.01313156 / (6)) + (18611318.156509932 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32140769421932874 = (355166.13088597404 - 469319.25807883526) / 355166.13088597404 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.058163905107143 (T) = (0 -440924.7506363231) / Math.Sqrt((12709204.964838311 / (6)) + (95079365.25673783 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3210033318196776 = (333780.1957160477 - 440924.7506363231) / 333780.1957160477 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Dictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.748163776809776 (T) = (0 -656561.9652148437) / Math.Sqrt((58608759.04845784 / (6)) + (300436053.9765758 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4392015457384698 = (456198.7632371217 - 656561.9652148437) / 456198.7632371217 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 5.53 ms 7.04 ms 1.27 0.04 False
SystemTextJson_SourceGen_ - Duration of single invocation 5.56 ms 7.08 ms 1.27 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<MyEventsListerViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.44769984365097 (T) = (0 -6988368.9389455775) / Math.Sqrt((12053481606.8467 / (6)) + (1991581966.0235996 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2673910132695846 = (5513980.189047698 - 6988368.9389455775) / 5513980.189047698 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;MyEventsListerViewModel&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -60.07469308599196 (T) = (0 -7020879.186319251) / Math.Sqrt((1555475087.264609 / (6)) + (1441685041.6972573 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.27064777687609193 = (5525433.02250148 - 7020879.186319251) / 5525433.02250148 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Reader.Tests.Perf_Base64

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 26.51 μs 61.51 μs 2.32 0.10 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 26.49 μs 61.53 μs 2.32 0.08 False
ReadBase64EncodedByteArray_NoEscaping - Duration of single invocation 3.68 μs 6.90 μs 1.87 0.04 False
ReadBase64EncodedByteArray_HeavyEscaping - Duration of single invocation 3.68 μs 7.02 μs 1.91 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Reader.Tests.Perf_Base64*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 1000)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -179.17074435707002 (T) = (0 -61006.184301600646) / Math.Sqrt((54884.154734159725 / (6)) + (111850.89275207692 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.3029220396158925 = (26490.772701873986 - 61006.184301600646) / 26490.772701873986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 1000)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -118.6158854615063 (T) = (0 -61258.47251298758) / Math.Sqrt((61569.97479684638 / (6)) + (303075.62423369446 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.3144977785216465 = (26467.285076469412 - 61258.47251298758) / 26467.285076469412 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_NoEscaping(NumberOfBytes: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -55.25495727362042 (T) = (0 -6921.797313835853) / Math.Sqrt((991.0585753978896 / (6)) + (13074.001803089159 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.8788858684941157 = (3683.990299731998 - 6921.797313835853) / 3683.990299731998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Reader.Tests.Perf_Base64.ReadBase64EncodedByteArray_HeavyEscaping(NumberOfBytes: 100)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -54.320575982073855 (T) = (0 -6913.211631537801) / Math.Sqrt((341.2551186761707 / (6)) + (13976.781721195592 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.8805312140777434 = (3676.201479552788 - 6913.211631537801) / 3676.201479552788 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<BinaryData>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 36.26 μs 71.82 μs 1.98 0.07 False
DeserializeFromStream - Duration of single invocation 34.51 μs 70.30 μs 2.04 0.06 False
DeserializeFromString - Duration of single invocation 36.33 μs 71.65 μs 1.97 0.07 False
DeserializeFromReader - Duration of single invocation 35.11 μs 107.07 μs 3.05 0.15 False
DeserializeFromUtf8Bytes - Duration of single invocation 29.62 μs 65.93 μs 2.23 0.10 False
DeserializeFromStream - Duration of single invocation 34.50 μs 70.41 μs 2.04 0.07 False
DeserializeFromReader - Duration of single invocation 35.16 μs 106.92 μs 3.04 0.13 False
DeserializeFromUtf8Bytes - Duration of single invocation 29.54 μs 65.63 μs 2.22 0.10 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<BinaryData>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -79.70334513133601 (T) = (0 -71773.41552960058) / Math.Sqrt((296223.5134392405 / (6)) + (583864.9234452051 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.9638667771505361 = (36546.98799566226 - 71773.41552960058) / 36546.98799566226 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -171.2374455957913 (T) = (0 -70074.09112567545) / Math.Sqrt((24661.967522748193 / (6)) + (157651.9369865335 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.039995429132166 = (34350.12163506937 - 70074.09112567545) / 34350.12163506937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -161.68942463300473 (T) = (0 -71193.27091992131) / Math.Sqrt((136203.82743855193 / (6)) + (94492.21763226122 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.9562377663072771 = (36392.95393745026 - 71193.27091992131) / 36392.95393745026 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -223.60929728636347 (T) = (0 -106149.63535999166) / Math.Sqrt((37159.624088950644 / (6)) + (379961.2690068963 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -2.0310340303549617 = (35020.93156887472 - 106149.63535999166) / 35020.93156887472 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -84.96580062422223 (T) = (0 -65618.52670493916) / Math.Sqrt((32404.586516658506 / (6)) + (699904.4450968188 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.2218809532759003 = (29532.87241073488 - 65618.52670493916) / 29532.87241073488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -195.29213156625536 (T) = (0 -69954.03595764082) / Math.Sqrt((35409.353387168056 / (6)) + (108240.03393653293 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.0277635412291874 = (34498.12294940275 - 69954.03595764082) / 34498.12294940275 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -208.54569809762774 (T) = (0 -106048.3322700848) / Math.Sqrt((185472.46876213085 / (6)) + (339389.3024132706 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -2.02184301477523 = (35093.9250489069 - 106048.3322700848) / 35093.9250489069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;BinaryData&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -197.59610742377276 (T) = (0 -65160.10210792817) / Math.Sqrt((39452.457823321085 / (6)) + (104412.56573413474 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -1.2133006875807513 = (29440.239400617287 - 65160.10210792817) / 29440.239400617287 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 20.59 μs 22.63 μs 1.10 0.03 False
DeserializeFromReader - Duration of single invocation 23.76 μs 26.64 μs 1.12 0.03 False
DeserializeFromReader - Duration of single invocation 23.86 μs 26.79 μs 1.12 0.07 False
DeserializeFromUtf8Bytes - Duration of single invocation 15.71 μs 17.75 μs 1.13 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 15.74 μs 17.54 μs 1.11 0.03 False
DeserializeFromString - Duration of single invocation 17.25 μs 18.87 μs 1.09 0.03 False
DeserializeFromStream - Duration of single invocation 20.77 μs 22.38 μs 1.08 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ClassRecord>.DeserializeFromStream(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -9.935227722459372 (T) = (0 -22171.575513717456) / Math.Sqrt((25964.072029499082 / (6)) + (92282.18185547634 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.0801142022470799 = (20527.065996902456 - 22171.575513717456) / 20527.065996902456 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.86833730705627 (T) = (0 -26641.484012662244) / Math.Sqrt((87461.30010556118 / (6)) + (30235.48882445871 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1319137085279523 = (23536.674052043552 - 26641.484012662244) / 23536.674052043552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.108070240168043 (T) = (0 -26716.98921445106) / Math.Sqrt((326919.5574449952 / (6)) + (8949.940956801498 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1099058939656749 = (24071.40043106872 - 26716.98921445106) / 24071.40043106872 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.474813981319596 (T) = (0 -17243.47151188823) / Math.Sqrt((53053.04387455371 / (6)) + (118188.19714031782 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10655752616807353 = (15582.986970050342 - 17243.47151188823) / 15582.986970050342 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.435610542869577 (T) = (0 -17489.283048032) / Math.Sqrt((22581.39013320791 / (6)) + (66487.54396193569 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11299188849220633 = (15713.75607393249 - 17489.283048032) / 15713.75607393249 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.963411111012153 (T) = (0 -18762.319855863654) / Math.Sqrt((21783.056030528853 / (6)) + (4989.226111547626 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.09346934017722194 = (17158.52394436847 - 18762.319855863654) / 17158.52394436847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ClassRecord&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.7035252387078 (T) = (0 -22202.878643483826) / Math.Sqrt((8733.669323876395 / (6)) + (14221.907242901887 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.07067376126831129 = (20737.29594081252 - 22202.878643483826) / 20737.29594081252 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 229.80 μs 262.26 μs 1.14 0.04 False
Parse - Duration of single invocation 20.90 ms 26.45 ms 1.27 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: ArrayOfStrings)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -9.063507638894317 (T) = (0 -261181.8667195921) / Math.Sqrt((27612039.772855688 / (6)) + (26494769.96402618 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13156469778239613 = (230814.78878887603 - 261181.8667195921) / 230814.78878887603 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateArray.Parse(TestCase: Json400KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.828257952942394 (T) = (0 -25474089.99605413) / Math.Sqrt((409441651305.28723 / (6)) + (424300943301.5247 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2662242783908339 = (20118150.023491554 - 25474089.99605413) / 20118150.023491554 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation 6.35 ms 7.22 ms 1.14 0.01 False
SystemTextJson_SourceGen_ - Duration of single invocation 6.34 ms 7.23 ms 1.14 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<CollectionsOfPrimitives>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.486753028901518 (T) = (0 -7196411.980028011) / Math.Sqrt((2079465400.186523 / (6)) + (5345774287.175269 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13958775904225998 = (6314925.658797939 - 7196411.980028011) / 6314925.658797939 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;CollectionsOfPrimitives&gt;.SystemTextJson_SourceGen_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -34.210547389703414 (T) = (0 -7172386.9892136855) / Math.Sqrt((1499596054.387225 / (6)) + (1416837831.565443 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13280728724924445 = (6331515.580757021 - 7172386.9892136855) / 6331515.580757021 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 7.67 μs 8.91 μs 1.16 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 7.53 μs 8.81 μs 1.17 0.02 False
DeserializeFromString - Duration of single invocation 8.82 μs 10.35 μs 1.17 0.03 False
DeserializeFromStream - Duration of single invocation 12.31 μs 13.56 μs 1.10 0.01 False
DeserializeFromReader - Duration of single invocation 12.82 μs 14.53 μs 1.13 0.02 False
DeserializeFromStream - Duration of single invocation 12.24 μs 13.49 μs 1.10 0.02 False
DeserializeFromString - Duration of single invocation 8.97 μs 10.39 μs 1.16 0.04 False
DeserializeFromReader - Duration of single invocation 12.23 μs 14.68 μs 1.20 0.06 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<LoginViewModel>.DeserializeFromUtf8Bytes(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -14.403084302474664 (T) = (0 -8768.834769344434) / Math.Sqrt((4824.835230334366 / (6)) + (23367.251186229332 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1546058692711745 = (7594.6563262142545 - 8768.834769344434) / 7594.6563262142545 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.375854946425084 (T) = (0 -8667.185808538205) / Math.Sqrt((26285.516675606777 / (6)) + (10914.593962121287 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13687761272892301 = (7623.675329250069 - 8667.185808538205) / 7623.675329250069 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.139280851009257 (T) = (0 -10436.054126026931) / Math.Sqrt((3225.356671226722 / (6)) + (20592.29492563813 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.18027059466996811 = (8842.08602091379 - 10436.054126026931) / 8842.08602091379 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.556271877170131 (T) = (0 -13565.46906054223) / Math.Sqrt((31956.67118056236 / (6)) + (2467.7626966168814 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.09697804624880653 = (12366.217452509924 - 13565.46906054223) / 12366.217452509924 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -14.632787021448106 (T) = (0 -14283.745023489522) / Math.Sqrt((66564.95293913381 / (6)) + (26606.261329529243 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1580350971072733 = (12334.466424350836 - 14283.745023489522) / 12334.466424350836 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.648800158158867 (T) = (0 -13558.878503203285) / Math.Sqrt((74872.99980553471 / (6)) + (59378.09857943183 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08820589709550546 = (12459.846559729956 - 13558.878503203285) / 12459.846559729956 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.3553214916224 (T) = (0 -10405.571879984125) / Math.Sqrt((3335.290791176073 / (6)) + (25330.202119218116 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1606703749335028 = (8965.139547548357 - 10405.571879984125) / 8965.139547548357 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;LoginViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -13.28539762537018 (T) = (0 -14434.229570426327) / Math.Sqrt((101684.33444091013 / (6)) + (39800.96232537641 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.17778927019378069 = (12255.358352900834 - 14434.229570426327) / 12255.358352900834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateObject

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
Parse - Duration of single invocation 444.75 μs 531.21 μs 1.19 0.03 False
Parse - Duration of single invocation 19.08 ms 22.14 ms 1.16 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Document.Tests.Perf_EnumerateObject*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateObject.Parse(TestCase: StringProperties)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -25.624912796640814 (T) = (0 -525749.8398344017) / Math.Sqrt((36309513.06760241 / (6)) + (15759712.242718449 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.18212139390506524 = (444751.1419259739 - 525749.8398344017) / 444751.1419259739 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Document.Tests.Perf_EnumerateObject.Parse(TestCase: ObjectProperties)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/2/2022 11:19:21 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -27.78764130228062 (T) = (0 -21928305.85357143) / Math.Sqrt((33906751547.654213 / (6)) + (23877413443.22192 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15821505975919162 = (18932844.698229548 - 21928305.85357143) / 18932844.698229548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 621.27 μs 770.12 μs 1.24 0.03 False
DeserializeFromStream - Duration of single invocation 617.79 μs 759.17 μs 1.23 0.03 False
DeserializeFromString - Duration of single invocation 623.31 μs 756.70 μs 1.21 0.04 False
DeserializeFromStream - Duration of single invocation 622.38 μs 744.17 μs 1.20 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 608.64 μs 697.30 μs 1.15 0.03 False
DeserializeFromReader - Duration of single invocation 739.61 μs 932.50 μs 1.26 0.03 False
DeserializeFromReader - Duration of single invocation 714.43 μs 916.47 μs 1.28 0.04 False
DeserializeFromUtf8Bytes - Duration of single invocation 597.26 μs 728.77 μs 1.22 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableDictionary<String, String>>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.446571099146485 (T) = (0 -752359.8842819941) / Math.Sqrt((9671360.047307137 / (6)) + (144035550.2439994 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.21188933750876404 = (620815.6644307082 - 752359.8842819941) / 620815.6644307082 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.482147410802813 (T) = (0 -744114.8993818681) / Math.Sqrt((26895359.239686467 / (6)) + (157056445.80568206 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2094365180015657 = (615257.509018679 - 744114.8993818681) / 615257.509018679 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.994168005070684 (T) = (0 -751082.084510046) / Math.Sqrt((76441072.06083438 / (6)) + (84210297.58907218 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.20515085761300486 = (623226.6108142552 - 751082.084510046) / 623226.6108142552 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.004252621555093 (T) = (0 -737705.4731300694) / Math.Sqrt((17431829.02162954 / (6)) + (121770909.37132443 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1967760415194037 = (616410.6295054945 - 737705.4731300694) / 616410.6295054945 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.028392219471378 (T) = (0 -692892.8924624742) / Math.Sqrt((89282664.26495636 / (6)) + (49236355.2805618 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15696883085442742 = (598886.3951942156 - 692892.8924624742) / 598886.3951942156 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.961851329393234 (T) = (0 -918239.2731765012) / Math.Sqrt((96181515.01451676 / (6)) + (147102190.4829999 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2712467278304995 = (722313.971846804 - 918239.2731765012) / 722313.971846804 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -33.45576551453998 (T) = (0 -908297.1734006165) / Math.Sqrt((104918258.50292596 / (6)) + (60145089.49255258 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2659131965683417 = (717503.5190902847 - 908297.1734006165) / 717503.5190902847 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.774053576162947 (T) = (0 -696366.6614687848) / Math.Sqrt((49805112.27945329 / (6)) + (472512441.8199427 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.16505647935335535 = (597710.6464875344 - 696366.6614687848) / 597710.6464875344 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromString - Duration of single invocation 926.46 μs 1.08 ms 1.17 0.03 False
DeserializeFromStream - Duration of single invocation 921.96 μs 1.04 ms 1.13 0.05 False
DeserializeFromReader - Duration of single invocation 1.02 ms 1.24 ms 1.21 0.06 False
DeserializeFromReader - Duration of single invocation 1.03 ms 1.23 ms 1.19 0.06 False
DeserializeFromUtf8Bytes - Duration of single invocation 915.40 μs 1.02 ms 1.11 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 904.39 μs 1.00 ms 1.11 0.06 False
DeserializeFromString - Duration of single invocation 927.49 μs 1.06 ms 1.14 0.06 False
DeserializeFromStream - Duration of single invocation 918.93 μs 1.06 ms 1.16 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<ImmutableSortedDictionary<String, String>>.DeserializeFromString(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.492023189264266 (T) = (0 -1068648.8333531746) / Math.Sqrt((106347387.36054114 / (6)) + (111632049.06797649 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.14881134235976942 = (930221.3461420626 - 1068648.8333531746) / 930221.3461420626 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.359938423124884 (T) = (0 -1035899.6960441468) / Math.Sqrt((93111274.31882866 / (6)) + (26390780.976858914 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12434030117574806 = (921340.0026316615 - 1035899.6960441468) / 921340.0026316615 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.991239197099643 (T) = (0 -1226881.7906058484) / Math.Sqrt((1001450703.0153763 / (6)) + (472300318.2340107 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.17818314732161378 = (1041333.6783800907 - 1226881.7906058484) / 1041333.6783800907 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.60569809174867 (T) = (0 -1233929.7767088881) / Math.Sqrt((498202496.3859003 / (6)) + (25906822.573643174 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.19855055118324116 = (1029518.3423767313 - 1233929.7767088881) / 1029518.3423767313 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.191515205796565 (T) = (0 -1003327.0391894889) / Math.Sqrt((211728704.74305472 / (6)) + (108724112.89397995 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10624250252185428 = (906968.4421835598 - 1003327.0391894889) / 906968.4421835598 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.67967387580762 (T) = (0 -1001734.7385248541) / Math.Sqrt((100941182.67406909 / (6)) + (33746340.32392571 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10955484896761372 = (902825.7949184928 - 1001734.7385248541) / 902825.7949184928 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -44.66271896695523 (T) = (0 -1057464.8612103174) / Math.Sqrt((34934721.026406236 / (6)) + (9561013.832151493 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13770584201527436 = (929471.2412983463 - 1057464.8612103174) / 929471.2412983463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;ImmutableSortedDictionary&lt;String, String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.604149688964554 (T) = (0 -1042587.8271978021) / Math.Sqrt((13039281.999953564 / (6)) + (195356359.31060684 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13713628770818245 = (916853.8885510935 - 1042587.8271978021) / 916853.8885510935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 5.57 ms 7.26 ms 1.30 0.01 False
DeserializeFromReader - Duration of single invocation 7.56 ms 10.23 ms 1.35 0.03 False
DeserializeFromString - Duration of single invocation 5.51 ms 7.04 ms 1.28 0.03 False
DeserializeFromReader - Duration of single invocation 7.52 ms 9.79 ms 1.30 0.03 False
DeserializeFromString - Duration of single invocation 5.54 ms 7.05 ms 1.27 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 5.17 ms 6.30 ms 1.22 0.02 False
DeserializeFromStream - Duration of single invocation 5.57 ms 7.16 ms 1.28 0.01 False
DeserializeFromUtf8Bytes - Duration of single invocation 5.21 ms 6.59 ms 1.26 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -27.90235447998369 (T) = (0 -7109798.463413221) / Math.Sqrt((1993328646.7880821 / (6)) + (11040333131.791779 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.27915702699958955 = (5558190.521839272 - 7109798.463413221) / 5558190.521839272 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.605166077088374 (T) = (0 -10034062.104218202) / Math.Sqrt((9751920264.570374 / (6)) + (21176152640.83772 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32525410068181126 = (7571425.056565317 - 10034062.104218202) / 7571425.056565317 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -63.00886583366264 (T) = (0 -6998237.761099773) / Math.Sqrt((2158889880.5595164 / (6)) + (859098973.5502253 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.27521759267981244 = (5487877.364045215 - 6998237.761099773) / 5487877.364045215 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -35.7464877358589 (T) = (0 -9778236.2545) / Math.Sqrt((5225577152.213197 / (6)) + (13125532090.101084 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.30816229127820355 = (7474788.349804594 - 9778236.2545) / 7474788.349804594 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.732127485449187 (T) = (0 -6998576.620789072) / Math.Sqrt((807497575.9689386 / (6)) + (9991117952.827147 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.266841354048057 = (5524430.188851874 - 6998576.620789072) / 5524430.188851874 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.929495672964478 (T) = (0 -6340399.732263049) / Math.Sqrt((10028506353.149284 / (6)) + (4501670205.5955 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.22386043580254975 = (5180655.854852694 - 6340399.732263049) / 5180655.854852694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.35737850744054 (T) = (0 -7029130.537326597) / Math.Sqrt((4296325132.902658 / (6)) + (7496961462.142899 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2699448931731145 = (5534988.624398846 - 7029130.537326597) / 5534988.624398846 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.27339485701281 (T) = (0 -6480811.426255159) / Math.Sqrt((1540718990.2790756 / (6)) + (10324625456.21163 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2492622968056113 = (5187710.733627937 - 6480811.426255159) / 5187710.733627937 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation 516.96 μs 744.55 μs 1.44 0.06 False
DeserializeFromUtf8Bytes - Duration of single invocation 501.01 μs 737.26 μs 1.47 0.03 False
DeserializeFromReader - Duration of single invocation 723.80 μs 1.25 ms 1.73 0.09 False
DeserializeFromString - Duration of single invocation 556.76 μs 845.41 μs 1.52 0.04 False
DeserializeFromReader - Duration of single invocation 725.45 μs 1.20 ms 1.65 0.03 False
DeserializeFromStream - Duration of single invocation 527.88 μs 831.08 μs 1.57 0.04 False
DeserializeFromString - Duration of single invocation 553.77 μs 865.60 μs 1.56 0.04 False
DeserializeFromStream - Duration of single invocation 530.27 μs 792.86 μs 1.50 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<IndexViewModel>.DeserializeFromUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.20939346218831 (T) = (0 -730738.0120519539) / Math.Sqrt((87287241.08285023 / (6)) + (192687135.0815707 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4631965972224668 = (499412.04991802707 - 730738.0120519539) / 499412.04991802707 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -46.30682065041783 (T) = (0 -740881.612684841) / Math.Sqrt((8232639.787104474 / (6)) + (104071071.17813228 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.48612259133280106 = (498533.3087631723 - 740881.612684841) / 498533.3087631723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.077930277132882 (T) = (0 -1171881.3055200565) / Math.Sqrt((144420960.90435976 / (6)) + (2980548950.4031777 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.6142360066035774 = (725966.5257905786 - 1171881.3055200565) / 725966.5257905786 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -164.87463508272128 (T) = (0 -842898.4058877663) / Math.Sqrt((6926673.999955435 / (6)) + (7375126.727330618 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.512165322243009 = (557411.5432282808 - 842898.4058877663) / 557411.5432282808 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -37.05957565234009 (T) = (0 -1168734.4656822344) / Math.Sqrt((24850163.410798818 / (6)) + (558851248.0163555 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.6137296479442885 = (724244.2791895604 - 1168734.4656822344) / 724244.2791895604 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.439835752326132 (T) = (0 -804987.0503310033) / Math.Sqrt((9263443.249048218 / (6)) + (519796989.3134864 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.5340850070506833 = (524734.3182622005 - 804987.0503310033) / 524734.3182622005 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -34.67726634046474 (T) = (0 -843775.0719326361) / Math.Sqrt((92342358.31963137 / (6)) + (212049255.05518177 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.5149285246229586 = (556973.5193563922 - 843775.0719326361) / 556973.5193563922 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;IndexViewModel&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -74.295136699751 (T) = (0 -786906.2262148008) / Math.Sqrt((9039945.237300728 / (6)) + (42410364.1221033 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4893049924983251 = (528371.4418325807 - 786906.2262148008) / 528371.4418325807 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Depth

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSpanEmptyLoop - Duration of single invocation 67.33 μs 80.18 μs 1.19 0.01 False
ReadSpanEmptyLoop - Duration of single invocation 568.11 μs 674.41 μs 1.19 0.05 False
ReadSpanEmptyLoop - Duration of single invocation 1.97 μs 2.28 μs 1.16 0.06 False
ReadSpanEmptyLoop - Duration of single invocation 65.72 μs 78.19 μs 1.19 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Depth*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Depth*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Depth*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Depth*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Depth.ReadSpanEmptyLoop(Depth: 65)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.00842035135216 (T) = (0 -78909.6628769428) / Math.Sqrt((1324683.8596779234 / (6)) + (718569.7303479211 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.17984596559329416 = (66881.32618842537 - 78909.6628769428) / 66881.32618842537 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Depth.ReadSpanEmptyLoop(Depth: 512)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -30.04298316320296 (T) = (0 -671115.0077256944) / Math.Sqrt((60395095.31484743 / (6)) + (11327160.103427717 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1915664666451297 = (563220.7908764227 - 671115.0077256944) / 563220.7908764227 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Depth.ReadSpanEmptyLoop(Depth: 1)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.313387661534282 (T) = (0 -2281.1176396383007) / Math.Sqrt((894.4089632463317 / (6)) + (671.4484744230386 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.16675664792007802 = (1955.0929010815935 - 2281.1176396383007) / 1955.0929010815935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Depth.ReadSpanEmptyLoop(Depth: 64)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.991905462883627 (T) = (0 -76089.23256901206) / Math.Sqrt((931376.6628440123 / (6)) + (2078982.4817925242 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1631684229360398 = (65415.490197842184 - 76089.23256901206) / 65415.490197842184 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<StructRecord>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 21.04 μs 24.44 μs 1.16 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 12.81 μs 14.00 μs 1.09 0.03 False
DeserializeFromString - Duration of single invocation 14.45 μs 16.25 μs 1.13 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 12.88 μs 14.30 μs 1.11 0.03 False
DeserializeFromReader - Duration of single invocation 20.72 μs 23.33 μs 1.13 0.03 False
DeserializeFromStream - Duration of single invocation 18.14 μs 19.72 μs 1.09 0.03 False
DeserializeFromString - Duration of single invocation 14.31 μs 15.89 μs 1.11 0.03 False
DeserializeFromStream - Duration of single invocation 17.94 μs 19.37 μs 1.08 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<StructRecord>.DeserializeFromReader(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.8104644016097815 (T) = (0 -23538.03176576631) / Math.Sqrt((242463.54239512663 / (6)) + (393118.7145017195 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12076595124343469 = (21001.73701712835 - 23538.03176576631) / 21001.73701712835 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.72347090920945 (T) = (0 -13945.39702253578) / Math.Sqrt((22051.815517511684 / (6)) + (6603.179003721371 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.09591268270518921 = (12724.9161749935 - 13945.39702253578) / 12724.9161749935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.068101881545424 (T) = (0 -16112.901271549154) / Math.Sqrt((8254.134923157206 / (6)) + (9657.67099330138 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12012441681837902 = (14384.921022716851 - 16112.901271549154) / 14384.921022716851 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.095787950319567 (T) = (0 -14129.421433332545) / Math.Sqrt((108697.64183626886 / (6)) + (22483.552172036547 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08386187828146609 = (13036.182669082953 - 14129.421433332545) / 13036.182669082953 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.157790245464874 (T) = (0 -23002.509026995715) / Math.Sqrt((60830.74150394379 / (6)) + (60810.955983549844 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11719323226997344 = (20589.55278511487 - 23002.509026995715) / 20589.55278511487 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.233246632793566 (T) = (0 -19758.7408025932) / Math.Sqrt((99526.38344454783 / (6)) + (165811.44513465939 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08225259305874331 = (18257.05101500341 - 19758.7408025932) / 18257.05101500341 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.10651873732514 (T) = (0 -15807.669763389393) / Math.Sqrt((88111.47947258879 / (6)) + (5911.219838976552 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10787033747535917 = (14268.519725342841 - 15807.669763389393) / 14268.519725342841 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;StructRecord&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.007695685723967 (T) = (0 -19304.574295690392) / Math.Sqrt((18875.247637106946 / (6)) + (13602.682982717153 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08163669549524696 = (17847.55858976423 - 19304.574295690392) / 17847.55858976423 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 9.52 μs 11.14 μs 1.17 0.04 False
SystemTextJson_Reflection_ - Duration of single invocation 9.53 μs 10.85 μs 1.14 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<LoginViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.489081187772836 (T) = (0 -10822.061467907592) / Math.Sqrt((2971.666851783245 / (6)) + (48515.484852285146 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1354377348312356 = (9531.180033853741 - 10822.061467907592) / 9531.180033853741 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;LoginViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.955300802308136 (T) = (0 -10830.201137377055) / Math.Sqrt((1521.6242978695682 / (6)) + (24695.747195718923 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13392974584227763 = (9551.033630687969 - 10830.201137377055) / 9551.033630687969 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Nullable<DateTimeOffset>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 3.98 μs 5.08 μs 1.28 0.06 False
DeserializeFromStream - Duration of single invocation 6.88 μs 7.45 μs 1.08 0.12 False
DeserializeFromString - Duration of single invocation 3.75 μs 4.36 μs 1.16 0.04 False
DeserializeFromString - Duration of single invocation 3.75 μs 4.33 μs 1.15 0.04 False
DeserializeFromStream - Duration of single invocation 6.91 μs 7.47 μs 1.08 0.04 False
DeserializeFromUtf8Bytes - Duration of single invocation 2.86 μs 3.20 μs 1.12 0.09 False
DeserializeFromUtf8Bytes - Duration of single invocation 2.71 μs 3.18 μs 1.18 0.12 False
DeserializeFromReader - Duration of single invocation 3.99 μs 5.02 μs 1.26 0.07 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Nullable<DateTimeOffset>>.DeserializeFromReader(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.490702225738595 (T) = (0 -5042.107168488691) / Math.Sqrt((124.3204426881222 / (6)) + (7628.890627467464 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2710892340731593 = (3966.76097423266 - 5042.107168488691) / 3966.76097423266 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.045788665147174 (T) = (0 -7455.451523106529) / Math.Sqrt((26660.248308019487 / (6)) + (506.4862560424022 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.06824728566498665 = (6979.143895943055 - 7455.451523106529) / 6979.143895943055 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/13/2023 1:32:26 AM, 2/15/2023 1:39:01 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.41370381454739 (T) = (0 -4330.408983472979) / Math.Sqrt((4248.031558743483 / (6)) + (479.2749293051993 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1569145358201077 = (3743.0673134409717 - 4330.408983472979) / 3743.0673134409717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/13/2023 1:32:26 AM, 2/15/2023 1:39:01 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.531719035534497 (T) = (0 -4324.143088040071) / Math.Sqrt((5768.41577485482 / (6)) + (239.91541215257766 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15870157538843585 = (3731.8867773097422 - 4324.143088040071) / 3731.8867773097422 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/11/2023 11:25:37 PM, 2/16/2023 1:35:06 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.91905713248213 (T) = (0 -7441.506876739413) / Math.Sqrt((426.3312373131497 / (6)) + (3065.958693209342 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08397105701849814 = (6865.0420401514675 - 7441.506876739413) / 6865.0420401514675 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/13/2023 1:32:26 AM, 2/15/2023 1:39:01 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.595359619575975 (T) = (0 -3164.2673330466096) / Math.Sqrt((4808.835217730875 / (6)) + (1519.207571002158 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1584913653793 = (2731.3689403378517 - 3164.2673330466096) / 2731.3689403378517 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 1/13/2023 1:32:26 AM, 2/15/2023 1:39:01 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.694158372039281 (T) = (0 -3255.191589598587) / Math.Sqrt((350.8845230683102 / (6)) + (10622.936528435317 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2064996792446521 = (2698.0459635402062 - 3255.191589598587) / 2698.0459635402062 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Nullable&lt;DateTimeOffset&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.402416999300883 (T) = (0 -4960.63978191889) / Math.Sqrt((11224.322491384335 / (6)) + (1651.8948910408587 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.22987912880649616 = (4033.4368359700616 - 4960.63978191889) / 4033.4368359700616 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 527.37 μs 785.48 μs 1.49 0.03 False
SystemTextJson_Reflection_ - Duration of single invocation 520.99 μs 784.46 μs 1.51 0.04 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromStream<IndexViewModel>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -95.83004550721141 (T) = (0 -782272.4525725447) / Math.Sqrt((18947871.441681597 / (6)) + (16241443.486005766 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4905930641701014 = (524806.1804232805 - 782272.4525725447) / 524806.1804232805 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromStream&lt;IndexViewModel&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -72.38020934268557 (T) = (0 -780158.9678565705) / Math.Sqrt((11879307.284877066 / (6)) + (43449875.083205394 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4980713057217478 = (520775.58983796294 - 780158.9678565705) / 520775.58983796294 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
SystemTextJson_SourceGen_ - Duration of single invocation 6.21 ms 7.09 ms 1.14 0.01 False
SystemTextJson_Reflection_ - Duration of single invocation 6.16 ms 7.14 ms 1.16 0.02 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<CollectionsOfPrimitives>.SystemTextJson_SourceGen_

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.520392407090277 (T) = (0 -6967814.472609891) / Math.Sqrt((2349655928.314376 / (6)) + (8445749820.25771 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12541624105566793 = (6191322.1245802445 - 6967814.472609891) / 6191322.1245802445 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### MicroBenchmarks.Serializers.Json_FromString&lt;CollectionsOfPrimitives&gt;.SystemTextJson_Reflection_

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.859749703739535 (T) = (0 -6946236.976279871) / Math.Sqrt((1428827626.9886432 / (6)) + (17457450702.293194 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.13100483245220965 = (6141651.0141864335 - 6946236.976279871) / 6141651.0141864335 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

performanceautofiler bot commented Mar 14, 2023 •
edited
Loading

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromReader - Duration of single invocation 257.35 μs 371.86 μs 1.44 0.06 False
DeserializeFromReader - Duration of single invocation 258.12 μs 378.15 μs 1.46 0.04 False
DeserializeFromStream - Duration of single invocation 212.03 μs 276.50 μs 1.30 0.03 False
DeserializeFromUtf8Bytes - Duration of single invocation 194.91 μs 254.39 μs 1.31 0.04 False
DeserializeFromString - Duration of single invocation 210.34 μs 277.60 μs 1.32 0.04 False
DeserializeFromUtf8Bytes - Duration of single invocation 194.24 μs 251.99 μs 1.30 0.03 False
DeserializeFromStream - Duration of single invocation 215.03 μs 273.02 μs 1.27 0.03 False
DeserializeFromString - Duration of single invocation 210.91 μs 275.24 μs 1.30 0.03 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<HashSet<String>>.DeserializeFromReader(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -29.775761446989964 (T) = (0 -363286.30388119933) / Math.Sqrt((5223934.64645818 / (6)) + (46333227.38741112 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4069569100647832 = (258207.12864935704 - 363286.30388119933) / 258207.12864935704 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -25.65915274810832 (T) = (0 -366462.730277578) / Math.Sqrt((6300177.96801832 / (6)) + (66224604.52905696 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.41602034659903253 = (258797.6445096572 - 366462.730277578) / 258797.6445096572 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromStream(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.89483764497566 (T) = (0 -271269.2296510438) / Math.Sqrt((1725089.1197998524 / (6)) + (15562889.957057202 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2783293766473045 = (212206.05159094915 - 271269.2296510438) / 212206.05159094915 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.93832229793866 (T) = (0 -248282.1185756087) / Math.Sqrt((1552347.847447059 / (6)) + (25061546.644842222 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2745439957253307 = (194800.7439588727 - 248282.1185756087) / 194800.7439588727 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -54.63737893358395 (T) = (0 -274546.7640251366) / Math.Sqrt((896787.9742712888 / (6)) + (4994982.229783908 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3077374260744244 = (209940.2820100309 - 274546.7640251366) / 209940.2820100309 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -27.638529692276006 (T) = (0 -248264.82022933397) / Math.Sqrt((731386.1585532353 / (6)) + (14204892.537113303 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2712329925295392 = (195294.50674130858 - 248264.82022933397) / 195294.50674130858 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.469697834129075 (T) = (0 -273205.27327574685) / Math.Sqrt((10119489.796842767 / (6)) + (10763807.005204068 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.27881508253844045 = (213639.389311421 - 273205.27327574685) / 213639.389311421 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;HashSet&lt;String&gt;&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/20/2022 11:23:14 PM, 2/28/2023 12:47:31 AM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -47.47697094972526 (T) = (0 -274701.50275684544) / Math.Sqrt((9260867.801890466 / (6)) + (914577.004717396 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2988272469990188 = (211499.6458470916 - 274701.50275684544) / 211499.6458470916 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Serialization.Tests.ReadJson<Location>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
DeserializeFromStream - Duration of single invocation 28.55 μs 30.91 μs 1.08 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 22.94 μs 24.94 μs 1.09 0.03 False
DeserializeFromString - Duration of single invocation 25.03 μs 27.53 μs 1.10 0.02 False
DeserializeFromUtf8Bytes - Duration of single invocation 23.36 μs 25.24 μs 1.08 0.02 False
DeserializeFromStream - Duration of single invocation 28.56 μs 30.81 μs 1.08 0.02 False
DeserializeFromString - Duration of single invocation 24.97 μs 27.79 μs 1.11 0.02 False
DeserializeFromReader - Duration of single invocation 35.26 μs 41.11 μs 1.17 0.01 False
DeserializeFromReader - Duration of single invocation 35.75 μs 39.04 μs 1.09 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<Location>.DeserializeFromStream(Mode: SourceGen)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.123931147573725 (T) = (0 -30890.616010795697) / Math.Sqrt((30089.2580310453 / (6)) + (67724.94473919486 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.0838059174273118 = (28501.98131794889 - 30890.616010795697) / 28501.98131794889 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromUtf8Bytes(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -26.807608660500208 (T) = (0 -24992.217137350613) / Math.Sqrt((23360.64837083441 / (6)) + (9310.213163995833 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.0924216001590454 = (22877.81304737292 - 24992.217137350613) / 22877.81304737292 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromString(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.77375427546841 (T) = (0 -27373.708767489683) / Math.Sqrt((55529.86847043014 / (6)) + (38966.17592234707 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.098287581900353 = (24923.990053793837 - 27373.708767489683) / 24923.990053793837 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromUtf8Bytes(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.118004132747737 (T) = (0 -24977.756574661875) / Math.Sqrt((75642.48045148281 / (6)) + (68198.2510306828 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.07499064739999103 = (23235.32454451946 - 24977.756574661875) / 23235.32454451946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromStream(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.137725785505976 (T) = (0 -30814.161661584058) / Math.Sqrt((46879.182355724006 / (6)) + (210198.64089614712 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.08793858152446564 = (28323.438643389178 - 30814.161661584058) / 28323.438643389178 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromString(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -25.69602589420953 (T) = (0 -27703.113374557004) / Math.Sqrt((28923.360870519467 / (6)) + (24136.69166424227 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10697567443891669 = (25025.945930201804 - 27703.113374557004) / 25025.945930201804 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromReader(Mode: Reflection)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.835217750019216 (T) = (0 -39514.39213142119) / Math.Sqrt((436807.87746739277 / (6)) + (1181006.6160812266 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.117246900026454 = (35367.64535259447 - 39514.39213142119) / 35367.64535259447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Serialization.Tests.ReadJson&lt;Location&gt;.DeserializeFromReader(Mode: SourceGen)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.459261575766003 (T) = (0 -39199.227826758666) / Math.Sqrt((106825.15857109621 / (6)) + (29458.113537502926 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10986702233198065 = (35318.85085151534 - 39199.227826758666) / 35318.85085151534 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler
Copy link
Author

Run Information

Architecture arm64
OS ubuntu 20.04
Baseline 795a2875908e26369011100f9882e188f2de8fab
Compare 2c4059efe04a46f115a802aaedaafed907b678ab
Diff Diff

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 164.02 μs 218.00 μs 1.33 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 1.61 ms 2.11 ms 1.31 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 41.72 μs 47.08 μs 1.13 0.01 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 3.87 μs 4.37 μs 1.13 0.10 False
ReadSpanEmptyLoop - Duration of single invocation 113.26 μs 169.36 μs 1.50 0.03 False
ReadReturnBytes - Duration of single invocation 18.76 μs 22.95 μs 1.22 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 1.30 ms 1.89 ms 1.46 0.01 False
ReadSpanEmptyLoop - Duration of single invocation 1.12 ms 1.61 ms 1.43 0.05 False
ReadReturnBytes - Duration of single invocation 23.96 μs 28.51 μs 1.19 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 28.79 μs 34.02 μs 1.18 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 113.42 μs 168.37 μs 1.48 0.03 False
ReadReturnBytes - Duration of single invocation 205.60 μs 225.44 μs 1.10 0.02 False
ReadReturnBytes - Duration of single invocation 305.60 μs 346.11 μs 1.13 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 78.65 μs 100.77 μs 1.28 0.01 False
ReadSpanEmptyLoop - Duration of single invocation 2.20 μs 2.64 μs 1.20 0.02 False
ReadReturnBytes - Duration of single invocation 2.58 μs 3.00 μs 1.16 0.04 False
ReadReturnBytes - Duration of single invocation 1.93 ms 2.36 ms 1.22 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 199.73 μs 224.21 μs 1.12 0.01 False
ReadSpanEmptyLoop - Duration of single invocation 15.04 μs 19.89 μs 1.32 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 1.12 ms 1.64 ms 1.46 0.07 False
ReadSpanEmptyLoop - Duration of single invocation 1.96 μs 2.36 μs 1.21 0.02 False
ReadReturnBytes - Duration of single invocation 1.42 ms 1.93 ms 1.36 0.02 False
ReadSpanEmptyLoop - Duration of single invocation 167.86 μs 212.09 μs 1.26 0.02 False
ReadSpanEmptyLoop - Duration of single invocation 1.62 ms 2.14 ms 1.32 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 4.25 μs 4.71 μs 1.11 0.09 False
ReadReturnBytes - Duration of single invocation 206.67 μs 232.29 μs 1.12 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 186.87 μs 253.08 μs 1.35 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 182.72 μs 202.06 μs 1.11 0.01 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 24.76 μs 31.00 μs 1.25 0.01 False
ReadSpanEmptyLoop - Duration of single invocation 181.46 μs 202.41 μs 1.12 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 20.13 μs 25.49 μs 1.27 0.01 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 19.07 μs 25.07 μs 1.31 0.06 False
ReadSpanEmptyLoop - Duration of single invocation 262.34 μs 288.57 μs 1.10 0.02 False
ReadSpanEmptyLoop - Duration of single invocation 77.24 μs 99.72 μs 1.29 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 261.71 μs 295.17 μs 1.13 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 166.19 μs 198.16 μs 1.19 0.02 False
ReadSpanEmptyLoop - Duration of single invocation 19.89 μs 24.47 μs 1.23 0.01 False
ReadReturnBytes - Duration of single invocation 35.39 μs 41.11 μs 1.16 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 132.45 μs 201.13 μs 1.52 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 195.28 μs 236.70 μs 1.21 0.01 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 2.29 μs 2.59 μs 1.13 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 34.04 μs 40.76 μs 1.20 0.07 False
ReadReturnBytes - Duration of single invocation 49.50 μs 55.33 μs 1.12 0.01 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 92.68 μs 120.33 μs 1.30 0.02 False
ReadSpanEmptyLoop - Duration of single invocation 172.53 μs 195.32 μs 1.13 0.02 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 1.83 ms 2.49 ms 1.36 0.02 False
ReadReturnBytes - Duration of single invocation 95.94 μs 117.47 μs 1.22 0.04 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 48.54 μs 54.54 μs 1.12 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 2.50 μs 3.01 μs 1.21 0.02 False
ReadSingleSpanSequenceEmptyLoop - Duration of single invocation 15.88 μs 20.60 μs 1.30 0.03 False
ReadSpanEmptyLoop - Duration of single invocation 41.46 μs 46.21 μs 1.11 0.05 False
ReadReturnBytes - Duration of single invocation 193.82 μs 247.77 μs 1.28 0.01 False
ReadMultiSpanSequenceEmptyLoop - Duration of single invocation 298.57 μs 336.29 μs 1.13 0.02 False
ReadReturnBytes - Duration of single invocation 2.35 μs 2.65 μs 1.13 0.03 False
ReadSpanEmptyLoop - Duration of single invocation 27.60 μs 34.30 μs 1.24 0.06 False
ReadReturnBytes - Duration of single invocation 142.75 μs 195.47 μs 1.37 0.01 False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Build files either built locally or downloaded from payload above)

  • Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
  • CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
  • Mono Runtime build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release
  • Dotnet SDK installed for dotnet commands
  • Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir/artifacts/bin/mono/$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
cp $RunDir/artifacts/bin/runtime/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/shared/Microsoft.NETCore.App/8.0.0 -rf
cp $RunDir/artifacts/bin/testhost/net8.0-$RunOS-Release-$RunArch/* $RunDir/artifacts/dotnet-mono -r
cp $RunDir/artifacts/bin/coreclr/$RunOS.$RunArch.Release/corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun" 

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Reader*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir/artifacts/dotnet-mono/shared/Microsoft.NETCore.App/8.0.0/corerun --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create mono dotnet
mkdir -p $RunDir/artifacts/dotnet-mono
$RunDir/build.sh -subset libs.pretest -configuration release -ci -arch $RunArch -testscope innerloop /p:RuntimeArtifactsPath=$RunDir\artifacts\bin\mono\$RunOS.$RunArch.Release /p:RuntimeFlavor=mono
xcopy $RunDir\artifacts\bin\runtime\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\shared\Microsoft.NETCore.App\8.0.0\ /e /y
xcopy $RunDir\artifacts\bin\testhost\net8.0-$RunOS-Release-$RunArch\ $RunDir\artifacts\dotnet-mono\ /e /y
xcopy $RunDir\artifacts\bin\coreclr\$RunOS.$RunArch.Release\corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# Use Interpreter
export MONO_ENV_OPTIONS="--interpreter"

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe" 

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter 'System.Text.Json.Tests.Perf_Reader*' --anyCategories Libraries Runtime " --category-exclusion-filter NoInterpreter NoMono --logBuildOutput --generateBinLog --corerun $RunDir\artifacts\dotnet-mono\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json4KB)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.925783145415906 (T) = (0 -210765.59598134877) / Math.Sqrt((4793979.418250054 / (6)) + (25522713.87044405 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2951571241153682 = (162733.61127925548 - 210765.59598134877) / 162733.61127925548 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.029181138082947 (T) = (0 -2069078.9806966146) / Math.Sqrt((2387877961.7070303 / (6)) + (935022370.2488106 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2804282294429945 = (1615927.338306728 - 2069078.9806966146) / 1615927.338306728 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.419803296493525 (T) = (0 -45914.82419026257) / Math.Sqrt((783725.75921057 / (6)) + (1154813.3577111098 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11687422671212083 = (41110.11167786336 - 45914.82419026257) / 41110.11167786336 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.845641680604242 (T) = (0 -4321.159521412422) / Math.Sqrt((2026.919512775039 / (6)) + (1951.5876530798912 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12615386481066895 = (3837.095139871409 - 4321.159521412422) / 3837.095139871409 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -28.34413272899565 (T) = (0 -166590.05978271604) / Math.Sqrt((2706398.862302588 / (6)) + (12906717.788906375 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.48432014245370986 = (112233.24067227723 - 166590.05978271604) / 112233.24067227723 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/16/2022 10:05:39 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -21.50554429632003 (T) = (0 -23027.0005219685) / Math.Sqrt((170787.19406798526 / (6)) + (39106.432367927846 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.22344015017123137 = (18821.517765904337 - 23027.0005219685) / 18821.517765904337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -46.17030799270261 (T) = (0 -1889705.5057002315) / Math.Sqrt((316611590.0201022 / (6)) + (481031619.6098915 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.47358855522001153 = (1282383.4027525359 - 1889705.5057002315) / 1282383.4027525359 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -25.371440618090745 (T) = (0 -1575657.1384753787) / Math.Sqrt((369516647.77564186 / (6)) + (1093927142.8991797 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4179314441494356 = (1111236.474074074 - 1575657.1384753787) / 1111236.474074074 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 12/17/2022 10:44:50 PM, 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.915961842248713 (T) = (0 -28058.575912819477) / Math.Sqrt((106456.11778061776 / (6)) + (161365.7612510041 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1818758787017433 = (23740.712894183962 - 28058.575912819477) / 23740.712894183962 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -9.53899625751046 (T) = (0 -32780.02010053462) / Math.Sqrt((165966.31895298528 / (6)) + (870778.9835705366 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1684186160275145 = (28055.030663567155 - 32780.02010053462) / 28055.030663567155 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -41.97572844513731 (T) = (0 -166286.18262490962) / Math.Sqrt((3887124.439814386 / (6)) + (3775730.3341829577 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.4673120695889259 = (113327.07340947277 - 166286.18262490962) / 113327.07340947277 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.894582366508434 (T) = (0 -222559.75613811795) / Math.Sqrt((22634483.504077423 / (6)) + (4171247.4822809673 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.09612766458647516 = (203041.82015338587 - 222559.75613811795) / 203041.82015338587 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -6.060822414456117 (T) = (0 -334195.12495840865) / Math.Sqrt((17892357.189471815 / (6)) + (83056035.75533107 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.09694175427095035 = (304660.7749748039 - 334195.12495840865) / 304660.7749748039 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.339574931950656 (T) = (0 -97790.26240186834) / Math.Sqrt((1806461.7288790494 / (6)) + (5707269.672960873 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.25975385750703583 = (77626.48379214999 - 97790.26240186834) / 77626.48379214999 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.250109348890659 (T) = (0 -2562.3652432366616) / Math.Sqrt((1507.5437279058467 / (6)) + (3717.605872379601 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.17766393671687875 = (2175.803438780751 - 2562.3652432366616) / 2175.803438780751 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.300254052357012 (T) = (0 -2949.9388328343452) / Math.Sqrt((1336.9530998659254 / (6)) + (1332.8626996050762 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1498179813325112 = (2565.570273492935 - 2949.9388328343452) / 2565.570273492935 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -18.991839223872166 (T) = (0 -2357800.8040979854) / Math.Sqrt((2760263883.524582 / (6)) + (240176646.1096722 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.22503297943047815 = (1924683.53398464 - 2357800.8040979854) / 1924683.53398464 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -9.969989362669633 (T) = (0 -219340.7969534987) / Math.Sqrt((13417536.279963106 / (6)) + (10771387.235011369 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11224289463277838 = (197205.84236765743 - 219340.7969534987) / 197205.84236765743 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.76418614329852 (T) = (0 -19127.74121246037) / Math.Sqrt((63275.08849248178 / (6)) + (256188.51023679433 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.29047298739284666 = (14822.27167815756 - 19127.74121246037) / 14822.27167815756 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.656865266221367 (T) = (0 -1598509.3617708334) / Math.Sqrt((228629766.6673581 / (6)) + (2049255355.9035368 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.43508410379977347 = (1113878.522895172 - 1598509.3617708334) / 1113878.522895172 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -13.551876691808248 (T) = (0 -2305.247440421501) / Math.Sqrt((988.1652591108519 / (6)) + (2146.82445165267 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.18440118460829416 = (1946.3400327346799 - 2305.247440421501) / 1946.3400327346799 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.26080358454623 (T) = (0 -1894789.6104166668) / Math.Sqrt((1230448055.1946769 / (6)) + (1625368690.7967727 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.33574098542843056 = (1418530.711482904 - 1894789.6104166668) / 1418530.711482904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.124404478435853 (T) = (0 -210994.85230720116) / Math.Sqrt((19710610.23455249 / (6)) + (14244443.360336957 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.31089070115827844 = (160955.3352699581 - 210994.85230720116) / 160955.3352699581 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.782156165028184 (T) = (0 -2068547.934382512) / Math.Sqrt((1205952821.9983397 / (6)) + (2423961005.4472 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2994943132772495 = (1591809.916555736 - 2068547.934382512) / 1591809.916555736 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.7162271457684 (T) = (0 -4668.933589402481) / Math.Sqrt((12600.506754861033 / (6)) + (1711.4279400997464 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10358585977752381 = (4230.693559578327 - 4668.933589402481) / 4230.693559578327 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.16626489328904 (T) = (0 -235761.64860093186) / Math.Sqrt((8547053.037876075 / (6)) + (8023166.617139328 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.14547494749634704 = (205819.99555401338 - 235761.64860093186) / 205819.99555401338 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -41.09769837166624 (T) = (0 -250485.65669223323) / Math.Sqrt((6418770.616145437 / (6)) + (5750870.051648724 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.3510048414264062 = (185406.9275042476 - 250485.65669223323) / 185406.9275042476 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.327108730761688 (T) = (0 -199047.41719866084) / Math.Sqrt((8365720.7864826005 / (6)) + (4986575.448240384 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10190233236374918 = (180639.80023680834 - 199047.41719866084) / 180639.80023680834 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -17.848053517023526 (T) = (0 -30191.856477970086) / Math.Sqrt((145417.86271261523 / (6)) + (322418.9276158611 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.23672211550592237 = (24412.805511785566 - 30191.856477970086) / 24412.805511785566 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -8.64520721042242 (T) = (0 -198121.04394558718) / Math.Sqrt((13130732.695793953 / (6)) + (10271699.970928714 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1051754344358848 = (179266.601276488 - 198121.04394558718) / 179266.601276488 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.527019846878376 (T) = (0 -24482.615943430817) / Math.Sqrt((109163.86597383698 / (6)) + (537022.9099814647 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.22523883291446753 = (19981.91314683863 - 24482.615943430817) / 19981.91314683863 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -44.65151653217141 (T) = (0 -24840.04193845048) / Math.Sqrt((57861.42469047895 / (6)) + (32070.294061778706 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.31386610440956525 = (18906.068019475453 - 24840.04193845048) / 18906.068019475453 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.241144953636523 (T) = (0 -285669.6640555457) / Math.Sqrt((32383801.776873935 / (6)) + (3770201.293335292 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10997645124341063 = (257365.51774187162 - 285669.6640555457) / 257365.51774187162 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -15.109239346040788 (T) = (0 -97067.47744793326) / Math.Sqrt((2244987.8084573117 / (6)) + (5450529.26834298 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.258077973844959 = (77155.37467941991 - 97067.47744793326) / 77155.37467941991 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.217713127078232 (T) = (0 -289485.95122559974) / Math.Sqrt((22405948.323056463 / (6)) + (59332466.012942694 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12036731839041802 = (258384.8586742885 - 289485.95122559974) / 258384.8586742885 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.315251549883053 (T) = (0 -193346.07163942503) / Math.Sqrt((7398998.037096657 / (6)) + (17399693.442689285 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.17716723282501315 = (164246.90243494578 - 193346.07163942503) / 164246.90243494578 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.926589141986184 (T) = (0 -23900.296464409395) / Math.Sqrt((76118.04498682599 / (6)) + (190264.45566790985 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.21042153185115892 = (19745.43234360468 - 23900.296464409395) / 19745.43234360468 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -20.100304824061173 (T) = (0 -40711.8533400742) / Math.Sqrt((47323.954682494616 / (6)) + (249908.22327162776 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15070282247478584 = (35379.989120489256 - 40711.8533400742) / 35379.989120489256 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -52.63675006391562 (T) = (0 -199299.52439652782) / Math.Sqrt((7483570.013547666 / (6)) + (1591218.7539069988 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.5122787919987859 = (131787.55494753234 - 199299.52439652782) / 131787.55494753234 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.266549570993238 (T) = (0 -234120.88096955308) / Math.Sqrt((5943169.46972017 / (6)) + (6811177.868598394 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.20496886793055613 = (194296.20731333763 - 234120.88096955308) / 194296.20731333763 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.1370298511748 (T) = (0 -2566.864983083492) / Math.Sqrt((2229.231912655142 / (6)) + (3193.1219597473514 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15616631967516564 = (2220.1520139461195 - 2566.864983083492) / 2220.1520139461195 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -22.783923706002874 (T) = (0 -40064.837708176085) / Math.Sqrt((90331.75035135771 / (6)) + (237104.78403735094 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.1834918673259304 = (33853.07395369058 - 40064.837708176085) / 33853.07395369058 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -12.990347248331515 (T) = (0 -54468.18483369486) / Math.Sqrt((388973.80530949024 / (6)) + (345480.4683254255 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10221606956412678 = (49416.97579788906 - 54468.18483369486) / 49416.97579788906 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -24.905938985345184 (T) = (0 -117998.70515801242) / Math.Sqrt((2651672.3959105588 / (6)) + (2529531.6671072836 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2800380699306044 = (92183.74666341726 - 117998.70515801242) / 92183.74666341726 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -13.140898820091774 (T) = (0 -194218.28503930083) / Math.Sqrt((20439237.93290316 / (6)) + (6893900.49121629 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.18098306288463803 = (164454.7590419361 - 194218.28503930083) / 164454.7590419361 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: Json40KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.1036259021088 (T) = (0 -2421464.9846141585) / Math.Sqrt((1838624266.868239 / (6)) + (2717608986.689089 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.32927180027427655 = (1821647.7503807147 - 2421464.9846141585) / 1821647.7503807147 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: DeepTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -16.000784390598657 (T) = (0 -115472.11269223918) / Math.Sqrt((1760716.398834462 / (6)) + (5231431.450984144 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2126333955269078 = (95224.25583707826 - 115472.11269223918) / 95224.25583707826 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.481717450166688 (T) = (0 -53691.19781690675) / Math.Sqrt((763931.9823174726 / (6)) + (645630.0262070111 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.11719344368801998 = (48058.99830531068 - 53691.19781690675) / 48058.99830531068 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -5.828060034340444 (T) = (0 -2828.9698399922345) / Math.Sqrt((1321.7283216639091 / (6)) + (15447.317230953564 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.15157624343625284 = (2456.606634703329 - 2828.9698399922345) / 2456.606634703329 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSingleSpanSequenceEmptyLoop(IsDataCompact: True, TestCase: Json400B)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -11.696178573987162 (T) = (0 -19672.12412933638) / Math.Sqrt((151990.10922550532 / (6)) + (456991.0577713331 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.28555869167027864 = (15302.392848184254 - 19672.12412933638) / 15302.392848184254 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: False, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.11955198389905 (T) = (0 -44851.89738614438) / Math.Sqrt((790589.8132945595 / (6)) + (828636.8072165186 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10182017440975134 = (40707.093977628145 - 44851.89738614438) / 40707.093977628145 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -19.174180040621522 (T) = (0 -241663.45869155787) / Math.Sqrt((8069134.939653014 / (6)) + (21057785.8379851 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.2562469034745083 = (192369.396512078 - 241663.45869155787) / 192369.396512078 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadMultiSpanSequenceEmptyLoop(IsDataCompact: False, TestCase: BroadTree)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.739074918605978 (T) = (0 -330029.33989546396) / Math.Sqrt((27502554.46952771 / (6)) + (18129388.15457329 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.10895103992630953 = (297604.9690321717 - 330029.33989546396) / 297604.9690321717 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: HelloWorld)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -10.33403666514997 (T) = (0 -2645.4684743570383) / Math.Sqrt((4723.897014059425 / (6)) + (206.14998501476518 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.12757181104862822 = (2346.164074372154 - 2645.4684743570383) / 2346.164074372154 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadSpanEmptyLoop(IsDataCompact: True, TestCase: LotsOfStrings)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -7.5143832953921885 (T) = (0 -33209.307860858695) / Math.Sqrt((433056.1150021475 / (6)) + (1879584.756251409 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.19989687940469647 = (27676.80159092904 - 33209.307860858695) / 27676.80159092904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: True, TestCase: Json4KB)

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because we could not find enough baseline builds for window checking.
IsChangePoint: Marked as a change because one of 3/8/2023 2:17:54 AM, 3/14/2023 1:07:29 AM falls between 3/4/2023 8:55:37 PM and 3/14/2023 1:07:29 AM.
IsRegressionStdDev: Marked as regression because -30.568578260930636 (T) = (0 -192375.7191626575) / Math.Sqrt((4927138.447346339 / (6)) + (7185487.023420531 / (4))) is less than -2.3060041352041716 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (6) + (4) - 2, .025) and -0.34604546269270037 = (142919.1840057311 - 192375.7191626575) / 142919.1840057311 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added arm64 PGO Applied if there were any profile guided optimization updates in the observed interval. untriaged labels Mar 14, 2023
@kotlarmilos
Copy link
Member

Most likely duplicate of #13992.

@kotlarmilos kotlarmilos added the LinkedFromMaster Is this issue linked from the master perf issue label Dec 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ampere arch-arm64 branch-refs/heads/main kind-micro_mono LinkedFromMaster Is this issue linked from the master perf issue mono-interpreter os-linux perf-regression PGO Applied if there were any profile guided optimization updates in the observed interval. runtime-mono
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@kotlarmilos