Include ImageSharp benchmarks in dotnet/performance

[kunalspathak]

Hi - ImageSharp is a great example of a .NET Core app and we would like to make sure that our changes doesn't adversely impact its performance. Will it be OK to include the snapshot of ImageSharp benchmarks code in https://github.com/dotnet/performance repo where .NET team can execute it daily and when we make changes in .NET, we can catch performance issues (see https://github.com/dotnet/perf-autofiling-issues) that would impact ImageSharp. Adding ImageSharp benchmarks to dotnet/performance repo will also have .NET team additional test coverage through our tools like superpmi.

We are planning to take a snapshot of the code and update it periodically (once a year maybe).

[JimBobSquarePants]

That would be awesome actually!

[antonfirsov]

Will it be OK to include the snapshot of ImageSharp benchmarks code

I think you need subset that is simpler/lighter. Our benchmarks are designed & maintained for selective local execution, and the Benchmarks project is not self-contained. You will face many issues trying to run the entire set regularly on CI.
I can create and share a minimal subset somewhere on GitHub.

[kunalspathak]

I can create and share a minimal subset somewhere on GitHub.

Thank you @antonfirsov. Let me know when things are ready on your end.

[antonfirsov]

Sorry, I have quite a lot on my plate, but still planning to help out on the benchmark front. It's hard to make a promise though, since my number one priority is to finish #1730. You probably have some timeline to finish this story, so let me know if it's an issue to wait for me.

I can create and share a minimal subset somewhere on GitHub.

Also reflecting on the discussions on #1794:

For clarity I mean by this that I likely don't have the capacity to push through and polish a proper PR to dotnet/performance, but I can create and share some benchmarks which are much better candidates to be copied there than our current benchmark project.

Also, we don't want to update the benchmark code or the version of ImageSharp frequently because that will change the measurement history even though coreclr didn't change.

I understand that your top priority is to detect runtime regressions, but for us it would be very nice for us to see and track the improvements we make with our NuGet releases. (We don't have any tracking infrastructure on our own -- lack of resources.) We don't release too frequently, approximately every half a year I would say. You definitely have to update when ImageSharp 2.0 is released, since it will be a massive improvement, making 1.0 discontinued.

[kunalspathak]

Thank you @antonfirsov . When is ImageSharp 2.0 release?

[antonfirsov]

"When it's done", meaning that we burn the backlog and author a sufficiently cool blogpost. My personal pessimistic estimation: January.

[kunalspathak]

I spent some time today trimming it and made sure that only ImageSharp related benchmarks stay. I have created a draft PR dotnet/performance#2149 here. I will appreciate if one of you take a look and sign off.