[Perf] Linux/arm64: 10 Regressions on 4/9/2023 11:24:15 AM #15660

performanceautofiler · 2023-04-11T11:06:48Z

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	0fc78e62bb0b8d824efe7421983702b97a60def6
Compare	86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_DateTime

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ToString - Duration of single invocation	118.62 ns	266.86 ns	2.25	0.02	False
ToString - Duration of single invocation	572.16 ns	922.83 ns	1.61	0.03	False
ToString - Duration of single invocation	557.22 ns	923.50 ns	1.66	0.01	False
ToString - Duration of single invocation	164.89 ns	290.44 ns	1.76	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_DateTime*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_DateTime* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_DateTime*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_DateTime* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_DateTime.ToString(format: "r")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 266.86328210102 > 123.97790129312402.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -115.43296577688585 (T) = (0 -269.2294900343203) / Math.Sqrt((0.11729856851795575 / (33)) + (6.827624352938642 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -1.2765709101472256 = (118.26097260326904 - 269.2294900343203) / 118.26097260326904 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: null)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 922.8342304196221 > 595.4106846242996.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -133.85899934841893 (T) = (0 -927.6621580309309) / Math.Sqrt((13.299929712690686 / (33)) + (27.469702634647145 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.6368745395884228 = (566.7277091768946 - 927.6621580309309) / 566.7277091768946 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: "G")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 923.496250404156 > 584.8545417232426.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -310.24587795931774 (T) = (0 -924.832486429385) / Math.Sqrt((16.17531980471892 / (33)) + (3.5592840743056318 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.650371260290728 = (560.3784485840279 - 924.832486429385) / 560.3784485840279 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTime.ToString(format: "o")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 290.44255076270014 > 172.83238807757928.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -115.67784679242914 (T) = (0 -289.6495109869555) / Math.Sqrt((0.36156286695675705 / (33)) + (4.617787930591928 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.7578932576189666 = (164.7708185531585 - 289.6495109869555) / 164.7708185531585 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	0fc78e62bb0b8d824efe7421983702b97a60def6
Compare	86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Tests.Perf_DateTimeOffset

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ToString - Duration of single invocation	144.78 ns	321.36 ns	2.22	0.01	False
ToString - Duration of single invocation	584.21 ns	960.69 ns	1.64	0.03	False
ToString - Duration of single invocation	211.48 ns	390.07 ns	1.84	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_DateTimeOffset*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_DateTimeOffset* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Tests.Perf_DateTimeOffset*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Tests.Perf_DateTimeOffset* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_DateTimeOffset.ToString(format: "r")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 321.36371776655085 > 151.2560982967572.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -164.43903440479932 (T) = (0 -320.0346532081729) / Math.Sqrt((1.7307860696241664 / (33)) + (4.372844854835347 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -1.2220531359347195 = (144.02655275547608 - 320.0346532081729) / 144.02655275547608 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTimeOffset.ToString(format: "G")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 960.6862730284049 > 617.1991478856135.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -145.56549969792312 (T) = (0 -954.6323107836247) / Math.Sqrt((27.67755744720513 / (33)) + (22.238043491203815 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.6278748437962305 = (586.42856631251 - 954.6323107836247) / 586.42856631251 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Tests.Perf_DateTimeOffset.ToString(format: "o")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 390.0690209009653 > 222.15119548655937.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -137.83927480394877 (T) = (0 -392.67823979461036) / Math.Sqrt((1.4789088943736353 / (33)) + (6.661098390480764 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.8485478120050565 = (212.42525470233076 - 392.67823979461036) / 212.42525470233076 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	arm64
OS	ubuntu 20.04
Queue	AmpereUbuntu
Baseline	0fc78e62bb0b8d824efe7421983702b97a60def6
Compare	86b48d7c6f081c12dcc9c048fb53de1b78c9966f
Diff	Diff
Configs	CompilationMode:tiered, LLVM:false, MonoAOT:true, MonoInterpreter:false, RunKind:micro_mono

Regressions in System.Globalization.Tests.Perf_DateTimeCultureInfo

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
ToString - Duration of single invocation	508.08 ns	794.21 ns	1.56	0.03	False
ToString - Duration of single invocation	505.50 ns	814.78 ns	1.61	0.01	False
ToString - Duration of single invocation	509.19 ns	816.52 ns	1.60	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

Repro Steps

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Libraries build extracted to runtime/artifacts or build instructions: Libraries README args: -subset libs+libs.tests -rc release -configuration Release -arch $RunArch -framework net8.0
CoreCLR product build extracted to runtime/artifacts/bin/coreclr/$RunOS.$RunArch.Release, build instructions: CoreCLR README args: -subset clr+libs -rc release -configuration Release -arch $RunArch -framework net8.0
AOT MONO build extracted to runtime/artifacts/bin/mono/$RunOS.$RunArch.Release, build instructions: MONO README args: -arch $RunArch -os $RunOS -s mono+libs+host+packs -c Release /p:CrossBuild=false /p:MonoLLVMUseCxx11Abi=false
Dotnet SDK installed for dotnet commands
Running commands from the runtime folder

Linux

# Set $RunDir to the runtime directory
RunDir=`pwd`

# Set the OS, arch, and OSId
RunOS='linux'
RunOSId='linux'
RunArch='x64'

# Create aot directory 
mkdir -p $RunDir/artifacts/bin/aot/sgen
mkdir -p $RunDir/artifacts/bin/aot/pack
cp -r $RunDir/artifacts/obj/mono/$RunOS.$RunArch.Release/mono/* $RunDir/artifacts/bin/aot/sgen
cp -r $RunDir/artifacts/bin/microsoft.netcore.app.runtime.$RunOS-$RunArch/Release/* $RunDir/artifacts/bin/aot/pack

# Create Core Root
$RunDir/src/tests/build.sh release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir/performance

# One line run:
python3 $RunDir/performance/scripts/benchmarks_ci.py --csproj $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.Perf_DateTimeCultureInfo*' --bdn-artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --packages $RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir/performance/artifacts/packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir/performance/src/benchmarks/micro/MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.Perf_DateTimeCultureInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir/artifacts/bin/aot/sgen/mini/mono-sgen --customruntimepack $RunDir/artifacts/bin/aot/pack --aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir/artifacts/BenchmarkDotNet.Artifacts --packages $RunDir/performance/artifacts/packages --buildTimeout 1200

Windows

# Set $RunDir to the runtime directory
$RunDir="FullPathHere"

# Set the OS, arch, and OSId
RunOS='windows'
RunOSId='win'
RunArch='x64'

# Create aot directory
mkdir $RunDir\artifacts\bin\aot\sgen
mkdir $RunDir\artifacts\bin\aot\pack
xcopy $RunDir\artifacts\obj\mono\$RunOS.$RunArch.Release\mono $RunDir\artifacts\bin\aot\sgen\ /e /y
xcopy $RunDir\artifacts\bin\microsoft.netcore.app.runtime.$RunOSId-$RunArch\Release $RunDir\artifacts\bin\aot\pack\ /e /y

# Create Core Root
$RunDir\src\tests\build.cmd release $RunArch generatelayoutonly /p:LibrariesConfiguration=Release

# Clone performance 
git clone --branch main --depth 1 --quiet https://github.com/dotnet/performance.git $RunDir\performance

# One line run:
python3 $RunDir\performance\scripts\benchmarks_ci.py --csproj $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --incremental no --architecture $RunArch -f net8.0 --filter 'System.Globalization.Tests.Perf_DateTimeCultureInfo*' --bdn-artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --bdn-arguments="--anyCategories Libraries Runtime  --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack --aotcompilermode llvm --logBuildOutput --generateBinLog"

# Individual Commands:
# Restore 
dotnet restore $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --packages $RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Build
dotnet build $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore /p:NuGetPackageRoot=$RunDir\performance\artifacts\packages /p:UseSharedCompilation=false /p:BuildInParallel=false /m:1

# Run
dotnet run --project $RunDir\performance\src\benchmarks\micro\MicroBenchmarks.csproj --configuration Release --framework net8.0 --no-restore --no-build -- --filter System.Globalization.Tests.Perf_DateTimeCultureInfo* --anyCategories Libraries Runtime " --category-exclusion-filter NoAOT NoWASM --runtimes monoaotllvm --aotcompilerpath $RunDir\artifacts\bin\aot\sgen\mini\mono-sgen.exe --customruntimepack $RunDir\artifacts\bin\aot\pack -aotcompilermode llvm --logBuildOutput --generateBinLog " --artifacts $RunDir\artifacts\BenchmarkDotNet.Artifacts --packages $RunDir\performance\artifacts\packages --buildTimeout 1200

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: fr)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 794.2125985856418 > 532.9962479231639.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -56.229325779110304 (T) = (0 -801.5711645353625) / Math.Sqrt((3.8506861078908265 / (33)) + (108.45507459290116 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.5774279044487235 = (508.15074481359204 - 801.5711645353625) / 508.15074481359204 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: ja)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 814.7793973320004 > 530.9891558676276.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -43.60726760162839 (T) = (0 -803.194462604415) / Math.Sqrt((2.5805205133949736 / (33)) + (186.5884675009153 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.5901283229554218 = (505.1129842851884 - 803.194462604415) / 505.1129842851884 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: )

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 816.5219419603717 > 534.7430840049018.
IsChangePoint: Marked as a change because one of 1/5/2023 4:05:11 PM, 4/9/2023 1:38:49 AM, 4/11/2023 12:48:48 AM falls between 4/1/2023 8:54:51 PM and 4/11/2023 12:48:48 AM.
IsRegressionStdDev: Marked as regression because -67.4892910113133 (T) = (0 -806.0607848835555) / Math.Sqrt((6.415171454067429 / (33)) + (76.32165789277646 / (4))) is less than -2.0301079282477414 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (33) + (4) - 2, .025) and -0.5812494473123944 = (509.76193936610986 - 806.0607848835555) / 509.76193936610986 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

kotlarmilos · 2023-04-11T12:28:57Z

Could be dotnet/runtime#84469. Assigning for accuracy tracking.

stephentoub · 2023-04-11T12:59:32Z

Same comment as here:
#15626 (comment)
"This change made a bunch of methods into generic methods, where the type parameter TChar is always either char or byte, and it's now invoking methods like TChar.CreateTruncating in a lot of places. There's a lot of IL in that code path, but it should resolve down to very simple asm assuming everything is properly getting inlined and evaluated for that TChar. Is it possible it's not? If those methods weren't inlined, it could definitely lead to significant regressions in these methods."

lewing · 2023-04-11T14:49:14Z

cc @vargaz

Re: dotnet/perf-autofiling-issues#15660.

…ave the aggressive inlining flag. Re: dotnet/perf-autofiling-issues#15660.

Re: dotnet/perf-autofiling-issues#15660.

…ave the aggressive inlining flag. (#84660) Re: dotnet/perf-autofiling-issues#15660.

vargaz · 2023-04-12T04:57:57Z

On wasm AOT, the hotspot appears to be the DateTimeFormat.FormatDigits () method which calls IBinaryInteger.CreateTruncating in its inner loop. Fixed a few issues in the mono code generator which got exposed by this, so the perf should be better now, but its still not back to the previous level.

kotlarmilos · 2023-05-10T09:40:12Z

@vargaz It has been mitigated, but not fully resolved. Do you think is there anything else we can do? Otherwise, I would close this issue.

vargaz · 2023-05-10T14:30:41Z

It can be closed i think.

performanceautofiler bot added ampere arch-arm64 branch-refs/heads/main kind-micro_mono mono-aot os-linux perf-regression runtime-mono untriaged labels Apr 11, 2023

kotlarmilos assigned stephentoub Apr 11, 2023

kotlarmilos removed the untriaged label Apr 11, 2023

kotlarmilos mentioned this issue Apr 11, 2023

.NET 8 per-preview performance overview dotnet/runtime#77490

Closed

kotlarmilos added the LinkedFromMaster Is this issue linked from the master perf issue label Apr 11, 2023

This was referenced Apr 11, 2023

[Perf] Linux/x64: 38 Regressions on 4/9/2023 11:24:15 AM #15699

Open

[Perf] Linux/x64: 38 Regressions on 4/9/2023 11:24:15 AM #15636

Open

kotlarmilos mentioned this issue Apr 11, 2023

[Perf] Linux/x64: 19 Regressions on 4/9/2023 11:24:15 AM #15697

Closed

vargaz added a commit to vargaz/runtime that referenced this issue Apr 11, 2023

[mono][jit] Intrinsify more calls to Type:op_Equality ().

77187ce

Re: dotnet/perf-autofiling-issues#15660.

vargaz mentioned this issue Apr 11, 2023

[mono][jit] Intrinsify more calls to Type:op_Equality (). dotnet/runtime#84653

Merged

This was referenced Apr 11, 2023

[Perf] Linux/x64: 19 Regressions on 4/9/2023 11:24:15 AM #15774

Closed

[Perf] Linux/arm64: 10 Regressions on 4/9/2023 11:24:15 AM #15795

Closed

[Perf] Linux/arm64: 2 Regressions on 4/9/2023 11:24:15 AM #15798

Closed

vargaz mentioned this issue Apr 11, 2023

[mono][inline] Allow inlining methods containing ctor calls if they h… dotnet/runtime#84660

Merged

vargaz added a commit to vargaz/runtime that referenced this issue Apr 11, 2023

[mono][inline] Allow inlining methods containing ctor calls if they h…

81437c4

…ave the aggressive inlining flag. Re: dotnet/perf-autofiling-issues#15660.

lewing pushed a commit to dotnet/runtime that referenced this issue Apr 12, 2023

[mono][jit] Intrinsify more calls to Type:op_Equality (). (#84653)

ae4c82b

Re: dotnet/perf-autofiling-issues#15660.

vargaz added a commit to dotnet/runtime that referenced this issue Apr 12, 2023

[mono][inline] Allow inlining methods containing ctor calls if they h…

8699942

…ave the aggressive inlining flag. (#84660) Re: dotnet/perf-autofiling-issues#15660.

kotlarmilos closed this as completed May 10, 2023

[Perf] Linux/arm64: 10 Regressions on 4/9/2023 11:24:15 AM #15660

[Perf] Linux/arm64: 10 Regressions on 4/9/2023 11:24:15 AM #15660

Comments

performanceautofiler bot commented Apr 11, 2023 • edited Loading

Run Information

Regressions in System.Tests.Perf_DateTime

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Tests.Perf_DateTime.ToString(format: "r")

Description of detection logic

System.Tests.Perf_DateTime.ToString(format: null)

Description of detection logic

System.Tests.Perf_DateTime.ToString(format: "G")

Description of detection logic

System.Tests.Perf_DateTime.ToString(format: "o")

Description of detection logic

Docs

Run Information

Regressions in System.Tests.Perf_DateTimeOffset

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Tests.Perf_DateTimeOffset.ToString(format: "r")

Description of detection logic

System.Tests.Perf_DateTimeOffset.ToString(format: "G")

Description of detection logic

System.Tests.Perf_DateTimeOffset.ToString(format: "o")

Description of detection logic

Docs

Run Information

Regressions in System.Globalization.Tests.Perf_DateTimeCultureInfo

Repro

Payloads

Prerequisites (Files either built locally (with build.(sh/cmd) or downloaded from payload above (if same system setup) (in this order))

Payloads

Histogram

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: fr)

Description of detection logic

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: ja)

Description of detection logic

System.Globalization.Tests.Perf_DateTimeCultureInfo.ToString(culturestring: )

Description of detection logic

Docs

kotlarmilos commented Apr 11, 2023

stephentoub commented Apr 11, 2023

lewing commented Apr 11, 2023

vargaz commented Apr 12, 2023

kotlarmilos commented May 10, 2023

vargaz commented May 10, 2023

performanceautofiler bot commented Apr 11, 2023 •

edited

Loading