Support executing benchmarks on arm64

[kunalspathak]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

Description

Because of existing roslyn issue dotnet/roslyn#42393, building and executing the benchmarks fail on arm64. In the past, we fixed this issue in dotnet/performance on arm64 in dotnet/performance#1243 and related change in dotnet/BenchmarkDotNet#1395. This PR does similar changes to make sure the benchmarks can be executed on arm64 platform.

[CLAassistant]

All committers have signed the CLA.

[brianpopow]

@kunalspathak could you provide some benchmark results on ARM? It Would interesting to see some real world example.

[codecov]

Codecov Report

Merging #1794 (cd7f406) into master (72904e1) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1794      +/-   ##
==========================================
+ Coverage   87.10%   87.11%   +0.01%     
==========================================
  Files         936      936              
  Lines       47631    47631              
  Branches     6015     6015              
==========================================
+ Hits        41488    41495       +7     
+ Misses       5150     5145       -5     
+ Partials      993      991       -2     

Flag	Coverage Δ
unittests	87.11% <ø> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ImageSharp/Formats/Webp/Lossless/Vp8LBitEntropy.cs	100.00% <0.00%> (+1.19%)	⬆️
...ImageSharp/Formats/Webp/BitReader/Vp8LBitReader.cs	95.65% <0.00%> (+8.69%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 72904e1...cd7f406. Read the comment docs.

[kunalspathak]

Sure, I filtered to Array2D and here are the results:

2.0, 1 CPU, 46 logical and 46 physical cores
.NET SDK=5.0.402
[Host] : .NET 5.0.11 (5.0.1121.47308), Arm64 RyuJIT
Job-LRNQQQ : .NET 5.0.11 (5.0.1121.47308), Arm64 RyuJIT

Method	Job	Runtime	Arguments	Count	Mean	Error	StdDev	Median	Ratio	RatioSD	Gen 0	Gen 1	Gen 2	Allocated
'Emulated 2D array access using flat array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	4	41.04 ns	0.238 ns	0.211 ns	41.12 ns	0.66	0.00	-	-	-	-
'Array access using 2D array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	4	62.38 ns	0.317 ns	0.297 ns	62.51 ns	1.00	0.00	-	-	-	-
'Array access using a jagged array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	4	54.61 ns	0.313 ns	0.261 ns	54.60 ns	0.87	0.01	-	-	-	-
'Array access using DenseMatrix'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	4	42.36 ns	0.009 ns	0.008 ns	42.36 ns	0.68	0.00	-	-	-	-
'Emulated 2D array access using flat array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	16	205.92 ns	0.030 ns	0.028 ns	205.92 ns	0.68	0.00	-	-	-	-
'Array access using 2D array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	16	300.83 ns	0.401 ns	0.356 ns	300.96 ns	1.00	0.00	-	-	-	-
'Array access using a jagged array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	16	216.49 ns	0.230 ns	0.215 ns	216.66 ns	0.72	0.00	-	-	-	-
'Array access using DenseMatrix'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	16	219.05 ns	0.082 ns	0.068 ns	219.05 ns	0.73	0.00	-	-	-	-
'Emulated 2D array access using flat array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	32	189.51 ns	0.172 ns	0.160 ns	189.47 ns	0.63	0.00	-	-	-	-
'Array access using 2D array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	32	301.97 ns	0.122 ns	0.114 ns	302.01 ns	1.00	0.00	-	-	-	-
'Array access using a jagged array'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	32	216.42 ns	0.409 ns	0.319 ns	216.30 ns	0.72	0.00	-	-	-	-
'Array access using DenseMatrix'	Job-LRNQQQ	.NET 5.0	/p:DebugType=portable	32	220.63 ns	0.074 ns	0.069 ns	220.67 ns	0.73	0.00	-	-	-	-

Now, there might be more changes needed on the code base:

1. When benchmarks are run, it always run on net3.1, netframework, etc. We might need a separate config for just net5.0 (and in future net6.0)
2. All the benchmarks should have MultiFramework or the config name that adds DebugType=portable to make sure they execute with the new parameter. For e.g. to execute Array2D, I had to do below change.

diff --git a/tests/ImageSharp.Benchmarks/General/Array2D.cs b/tests/ImageSharp.Benchmarks/General/Array2D.cs
index cd4eec0d4..763a47d9d 100644
--- a/tests/ImageSharp.Benchmarks/General/Array2D.cs
+++ b/tests/ImageSharp.Benchmarks/General/Array2D.cs
@@ -18,6 +18,7 @@ namespace SixLabors.ImageSharp.Benchmarks.General

      *
      */
+    [Config(typeof(Config.MultiFramework))]
     public class Array2D
     {
         private float[] flatArray;

[kunalspathak]

Actually, one more thing I realized is ImageSharp doesn't support Arm64 intrinsic yet, yes?

[brianpopow]

Actually, one more thing I realized is ImageSharp doesn't support Arm64 intrinsic yet, yes?

yes, we are not able to run the tests in the CI on ARM, that is mainly the reasons we dont have support for that.

[kunalspathak]

yes, we are not able to run the tests in the CI on ARM, that is mainly the reasons we dont have support for that.

Makes sense. Let me know what else is needed in this PR. Thanks!

[kunalspathak]

Related - #1795

[brianpopow]

I tried tried to get the tests to run on ARM for a while now and i think im pretty close to it. I was able to to run all the tests on
odroid-n2 (with some very minor exceptions). I think it should be possible to run the tests on docker now (with QEMU), but the tradeoff would be that it take a long time to execute the tests (my very rough estimate would be more then 30 min)

[brianpopow]

@kunalspathak thanks for the contribution!

[brianpopow]

Something worth noting is, that some of the benchmarks use Magick.NET as the baseline. This will not work out of the box on ARM. Magick.Native needs to be compiled for ARM for this to work.

[kunalspathak]

Something worth noting is, that some of the benchmarks use Magick.NET as the baseline

Does it have to be that away? Can we turn the baseline off using a switch and just run the ImageSharp part of benchmarks and get the numbers?

[brianpopow]

Does it have to be that away? Can we turn the baseline off using a switch and just run the ImageSharp part of benchmarks and get the numbers?

The normal use case of those benchmarks is actually to compare our implementation against other implementations. Of course an optional switch to turn the baseline off is a possibility. I hope some day Magick.NET will support ARM.

[kunalspathak]

Of course an optional switch to turn the baseline off is a possibility.

Great. Yes, at least that will give us a way to run and measure the coreclr build over build performance on just ImageSharp benchmarks.

[brianpopow]

@kunalspathak: I kind of thought the idea was that @antonfirsov would create a separate repo for that, which is more suitable to run in a CI. Did I misunderstood that?

[kunalspathak]

I was actually thinking if you or @antonfirsov can suggest the files that we can copy into dotnet/performance. We have our internal system that clones that repo, runs benchmarks, convert results in parse-able format, does regression analysis and automatically open GH issues. It might be hard to extend this for a different repo. So yes, the goal would be point me (in a different repo or different branch of this repo), what minimal *.cs, .csproj I should copy to dotnet/performance repo that will give us coverage for ImageSharp. Also, we don't want to update the benchmark code or the version of ImageSharp frequently because that will change the measurement history even though coreclr didn't change.

Just to give you an idea, here is how are infrastructure tracks the measurement history (currently of https://github.com/dotnet/performance/tree/main/src/benchmarks/micro):

https://pvscmdupload.blob.core.windows.net/reports/allTestHistory%2frefs%2fheads%2fmain_x64_ubuntu%2018.04%2fBenchstone.BenchI.BubbleSort2.Test.html

[kesiasimon]

Hi . I am trying to test the performance (both memory and cpu usage for file read &write ,Network IO) of Arm64 machine by running a simple performance testing console application using benchmarkdotnet. (I need to test it for both x64 as well as ARM64 platform in .net4.7.2 and .net6) .Now i've build a benchmark application with .net6 and platform as ARM64. but m not able to run it on ARM64 machine. Could you plz help to solve this issue.

[kunalspathak]

but m not able to run it on ARM64 machine. Could you plz help to solve this issue.

Can you give details of steps you are using to run benchmarks and what errors are you getting?

[kesiasimon]

I've created a simple benchmark application on .net 6 and it is running fine on visual studio.but if i take the release folder out and try to test the same then the application is not running. it is showing error like unable to find the .sln or .csproj file. and i try to publish the application and tried with the setup .which is also not working and giving the same error. now i've tried it within my pc (instead of ARM pc) that is also giving the same error. the application is only running inside the visual studio. but for testing i need a build file. but for .net 4.7.2 it is working fine.

Error: Most probably the name of output exe is different than the name of the .(c/f)sproj