Add ImageSharp benchmark #2149

kunalspathak · 2021-11-20T01:09:38Z

This is a snapshot taken from SixLabors/ImageSharp@413c90c. I deleted System.Drawing* related benchmarks and most of the other 3rd party baseline benchmarks to keep things simple. I also copied few TestUtilities related code and the input images.
I ran it on Windows/x64 and Windows/arm64 and they seem to be running fine. It needs some more refining to get 100% pass rate.

Relevant discussion: SixLabors/ImageSharp#1795 (reply in thread)

FYI - @antonfirsov , @JimBobSquarePants

JimBobSquarePants · 2021-11-22T05:13:08Z

Will have a look ASAP. A little concerned about how to maintain this.

Ideally I would say that there would be a simple set of benchmarks that use a reference to ImageSharp via NuGet. That way anyone can PR to update the references when required.

danmoseley · 2021-11-22T21:47:08Z

if you go ahead with this, there probably should be an entry in THIRD_PARTY_NOTICES.TXT

I agree with @JimBobSquarePants isn't a NuGet reference to a fixed package version sufficient? There is a lot of code here that requires building, etc.

kunalspathak · 2021-11-23T06:47:24Z

that use a reference to ImageSharp via NuGet

That's what I have currently. It reference to the ImageSharp version 1.0.2 via nuget. The reason there are so many files are because I copied every single benchmark file present in the original repo modulo the other baseline benchmarks. Perhaps, if there is a subset of benchmarks that you think can represent a good coverage of ImageSharp, feel free to point out and I am happy to trim it to that subset.

antonfirsov

The main challenge here is that these benchmarks were developed with local execution / sandboxing in mind, many of them are garbage for dotnet/runtime performance tracking. Tried to filter those out in my review.

I'm quite sure that most of the test images are unused by the benchmarks, those are better to be deleted.

With aggressive filtering this approach (of starting with the original IS benchmark project) can work out, but there is still a lot to delete. I wonder if it would save time, if you instead tried the other way: start with an (almost) empty project (containing only the TestEnfironment infra bits), and include the benchmarks which bring value one-by-one. Less can be more here in the end of the day.

What I would include:

Codecs, but only individual encode/decode benchmarks, not the ones that work on multiple files
Individual processors, stuff like resize, rotate, blur etc.. (I couldn't find Resize, you will have to bring that here, that's our most optimized operation relying heavily on SIMD)
Some of the other stuff, eg. color conversion

If you manage to get rid of the noisy stuff, I can do another review.

src/benchmarks/real-world/ImageSharp/Codecs/ImageBenchmarkTests.cs

src/benchmarks/real-world/ImageSharp/Codecs/Jpeg/DecodeJpeg_Aggregate.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeBmpMultiple.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeGifMultiple.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeIndexedPng.cs

src/benchmarks/real-world/ImageSharp/Processing/Diffuse.cs

src/benchmarks/real-world/ImageSharp/Processing/Skew.cs

src/benchmarks/real-world/ImageSharp/Program.cs

src/benchmarks/real-world/ImageSharp/TestUtilities/TestEnvironment.cs

src/benchmarks/real-world/ImageSharp/TestUtilities/TestImages.cs

kunalspathak · 2021-12-02T19:01:45Z

if you go ahead with this, there probably should be an entry in THIRD_PARTY_NOTICES.TXT

Done.

kunalspathak · 2021-12-02T19:03:54Z

If you manage to get rid of the noisy stuff, I can do another review.

@antonfirsov - Thanks for taking time and providing feedback. I have deleted all the benchmarks that you have suggested. Do you mind doing another round of review?

antonfirsov

If you pull in the new Jpeg benchmarks, add Resize benchmark, fix the memoryStream issues this should be mostly good to go.

Any insights whether the variance of execution times is within acceptable range for these benchmarks?

src/benchmarks/real-world/ImageSharp/Codecs/DecodeFilteredPng.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeBmp.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeGif.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeIndexedPng.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodePng.cs

src/benchmarks/real-world/ImageSharp/Codecs/EncodeTga.cs

src/benchmarks/real-world/ImageSharp/Codecs/Jpeg/DecodeJpeg_ImageSpecific.cs

antonfirsov · 2021-12-04T19:28:28Z

Regarding the resize benchmark:

We don't need the maxDegreeOfParallelism stuff it's outdated, resize implementation is not parallel anymore.

The subclasses exercising different scenarios are valuable, except Resize_Bicubic_Rgba32_CompareWorkBufferSizes, which can be trimmed.

danmoseley · 2022-02-05T00:52:22Z

@kunalspathak are you still planning to continue with this?

kunalspathak · 2022-02-05T00:54:01Z

@kunalspathak are you still planning to continue with this?

Yes. I have been busy with Arm64 investigation work. Will get to this soon. I don't think much work is needed here.

kunalspathak · 2022-08-12T23:23:27Z

Ping to myself.

kunalspathak · 2022-08-17T06:28:43Z

@antonfirsov , @JimBobSquarePants - I will resume working on this project now. Wondering what should be the next steps? Should I update the ImageSharp code to more latest or should we get this in as-is (after addressing the review comments) and then do another PR to update the code base to the latest?

JimBobSquarePants · 2022-08-17T11:52:05Z

If you are going to continue the snapshot approach I would recommend updating to the latest V2.x version before continuing. V1 numbers won't be beneficial and I can imagine you don't want to go through the PR process multiple times.

However, for ease of maintenance I would suggest that the benchmarks are performed using NuGet packages.

kunalspathak · 2022-08-17T15:20:03Z

If you are going to continue the snapshot approach I would recommend updating to the latest V2.x version before continuing. V1 numbers won't be beneficial and I can imagine you don't want to go through the PR process multiple times.

I agree. I will update the version in https://github.com/dotnet/performance/pull/2149/files#diff-99843982e8a5e1acef45cd069c3e3230ec1a014e7fe01aea4c05f2339b40afbc.

However, for ease of maintenance I would suggest that the benchmarks are performed using NuGet packages.

As part of this PR, ImageSharp will be consumed via nuget packages, if that's what you meant by "benchmarks are performed using NuGet packages.". My question was more around if I should take the latest benchmarks source code from ImageSharp repo or not.

kunalspathak · 2022-08-22T15:11:40Z

Resize benchmarks can be based on this code, but have to be heavily sanitized, see [Add ImageSharp benchmark Add ImageSharp benchmark #2149 (comment)]
(Add ImageSharp benchmark #2149 (comment)).

Done.

The jpeg decoder benchmark DecodeJpeg_ImageSpecific should be replaced with DecodeJpeg from Unified jpeg benchmark SixLabors/ImageSharp#1844

Done.

I would consider removing the benchmarks under the folder Color, it's an unfinished niche feature

Done.

would recommend updating to the latest V2.x version before continuing

Could you confirm the exact version that I will put in ImageSharp.Benchmarks.csproj?

antonfirsov · 2022-08-22T15:24:30Z

Could you confirm the exact version that I will put in ImageSharp.Benchmarks.csproj?

2.1.3

kunalspathak · 2022-08-22T15:27:54Z

Could you confirm the exact version that I will put in ImageSharp.Benchmarks.csproj?

2.1.3

I tried that and getting these errors:

Error	NU1102	Unable to find package SixLabors.ImageSharp with version (>= 2.1.3)
  - Found 13 version(s) in dotnet-public [ Nearest version: 1.0.2 ]
  - Found 0 version(s) in benchmark-dotnet-prerelease
  - Found 0 version(s) in dotnet-eng
  - Found 0 version(s) in dotnet-tools
  - Found 0 version(s) in dotnet3.1-transport
  - Found 0 version(s) in dotnet3.1
  - Found 0 version(s) in dotnet5-transport
  - Found 0 version(s) in dotnet5
  - Found 0 version(s) in dotnet6
  - Found 0 version(s) in dotnet7	ImageSharp.Benchmarks	D:\git\performance\src\benchmarks\real-world\ImageSharp\ImageSharp.Benchmarks.csproj	1

Also I am getting these errors for

this.Configuration.WorkingBufferSizeHintInBytes = this.WorkingBufferSizeHintInKilobytes * 1024;

Error	CS0281	Friend access was granted by 'SixLabors.ImageSharp, Version=1.0.0.0, Culture=neutral, PublicKeyToken=d998eea7b14cab13', but the public key of the output assembly ('') does not match that specified by the InternalsVisibleTo attribute in the granting assembly.	ImageSharp.Benchmarks (net5.0)	D:\git\performance\src\benchmarks\real-world\ImageSharp\Processing\Resize.cs	79	N/A

kunalspathak · 2022-08-27T03:29:54Z

@antonfirsov, @JimBobSquarePants - This is ready for the review. Let me know how I can upgrade to the latest ImageSharp nuget version.

kunalspathak · 2022-09-02T17:44:30Z

ping @antonfirsov, @JimBobSquarePants

JimBobSquarePants · 2022-09-02T21:16:26Z

Apologies, I’ll review over the weekend

kunalspathak · 2022-09-07T14:01:20Z

Apologies, I’ll review over the weekend

I can understand things can be busy so just sending a friendly remainder in case it was forgotten. :)

antonfirsov

Error NU1102 Unable to find package SixLabors.ImageSharp with version (>= 2.1.3)

I don't see the cause of this issue, I would have to clone the repository and try it myself to figure out. Any chance you can try to investigate? I will be on vacation the rest of the week.

antonfirsov · 2022-09-07T14:24:31Z

src/benchmarks/real-world/ImageSharp/Processing/Resize.cs

+    /// Is it worth to set a larger working buffer limit for resize?
+    /// Conclusion: It doesn't really have an effect.
+    /// </summary>
+    public class Resize_Bicubic_Rgba32_CompareWorkBufferSizes : Resize_Bicubic_Rgba32


This benchmark is the cause of CS0281, it's not needed for non-regression purposes.

antonfirsov · 2022-09-07T14:25:42Z

src/benchmarks/real-world/ImageSharp/Processing/Resize.cs

+            => ctx.Resize(this.DestSize, this.DestSize, KnownResamplers.Bicubic, true);
+    }
+
+    public class Resize_Bicubic_Compare_Rgba32_Rgb24


In general, if a benchmark is about comparing A vs B, it's safe to trim it.

@kunalspathak this should be trimmed away,

kunalspathak · 2022-10-28T14:31:44Z

I think I got this working with just 1 problem remaining. For some reason, I get compilation error while trying to invoke Decode() at https://github.com/kunalspathak/performance/blob/dae25ad21d40e7cc3781d45f9870a39ace45e86b/src/benchmarks/real-world/ImageSharp/Codecs/Jpeg/DecodeJpeg.cs#L27. It resolves to ImageSharp 2.0 version code that navigates to this definition:

I tried clearing my nuget packages, but it doesn't help.

antonfirsov · 2022-10-28T21:27:35Z

The code you took is probably from the time before we changed IImageDecoder to take a CancellationToken. What if you change it like this?

- using Image img = this.decoder.Decode(Configuration.Default, this.preloadedImageStream);
+ using Image img = this.decoder.Decode(Configuration.Default, this.preloadedImageStream, default);

kunalspathak · 2022-10-28T21:31:41Z

What if you change it like this?

that works. Wanted to make sure if default is fine with benchmark code. Any other comments or should it be good to go?

kunalspathak · 2022-11-01T13:43:51Z

Ping @antonfirsov and @JimBobSquarePants

JimBobSquarePants · 2022-11-03T01:59:50Z

I cannot see anything out of the ordinary 👍

kunalspathak · 2022-11-03T05:06:41Z

I cannot see anything out of the ordinary 👍

Thanks for confirming @JimBobSquarePants and thank you for the valuable feedback. @sblom - This one is another "real-world" benchmark we would like to merge.

antonfirsov

Good in general, but noticed one point from previous review that was not addressed. Might worth a final self-review if there's anything left that can be trimmed away.

antonfirsov · 2022-11-03T13:32:59Z

src/benchmarks/real-world/ImageSharp/Processing/Resize.cs

+            => ctx.Resize(this.DestSize, this.DestSize, KnownResamplers.Bicubic, true);
+    }
+
+    public class Resize_Bicubic_Compare_Rgba32_Rgb24


@kunalspathak this should be trimmed away,

kunalspathak · 2022-11-04T04:59:15Z

one point from previous review that was not addressed.
Fixed.

Might worth a final self-review if there's anything left that can be trimmed away.

We are now with following benchmarks:

SixLabors
 └─ImageSharp
    └─Benchmarks
       ├─DetectEdges
       │  └─ImageProcessorCoreDetectEdges
       ├─Resize_Bicubic_Rgb24
       │  └─ImageSharp_P1
       ├─Resize_Bicubic_Rgba32
       │  └─ImageSharp_P1
       ├─Codecs
       │  ├─DecodeBmp
       │  │  └─BmpImageSharp
       │  ├─DecodeFilteredPng
       │  │  ├─PngFilter0
       │  │  ├─PngFilter1
       │  │  ├─PngFilter2
       │  │  ├─PngFilter3
       │  │  └─PngFilter4
       │  ├─DecodeGif
       │  │  └─GifImageSharp
       │  ├─DecodePng
       │  │  └─PngImageSharp
       │  ├─DecodeTga
       │  │  └─TgaImageSharp
       │  ├─EncodeBmp
       │  │  └─BmpImageSharp
       │  ├─EncodeGif
       │  │  └─GifImageSharp
       │  ├─EncodeIndexedPng
       │  │  ├─PngCoreOctree
       │  │  ├─PngCoreOctreeNoDither
       │  │  ├─PngCorePalette
       │  │  ├─PngCorePaletteNoDither
       │  │  ├─PngCoreWu
       │  │  └─PngCoreWuNoDither
       │  ├─EncodePng
       │  │  └─PngCore
       │  ├─EncodeTga
       │  │  └─ImageSharpTga
       │  └─Jpeg
       │     └─DecodeJpeg
       │        ├─JpegBaselineInterleaved444
       │        ├─JpegBaselineInterleaved420
       │        ├─JpegBaseline400
       │        └─JpegProgressiveNonInterleaved420
       ├─Processing
       │  ├─BokehBlur
       │  │  └─Blur
       │  ├─Crop
       │  │  └─CropImageSharp
       │  ├─Diffuse
       │  │  ├─DoDiffuse
       │  │  └─DoDither
       │  ├─HistogramEqualization
       │  │  ├─GlobalHistogramEqualization
       │  │  └─AdaptiveHistogramEqualization
       │  └─Rotate
       │     └─DoRotate
       └─Samplers
          └─GaussianBlur
             └─Blur

antonfirsov reviewed Nov 23, 2021

View reviewed changes

kunalspathak mentioned this pull request Dec 2, 2021

ML.NET rolling and fixed version CI legs #988

Closed

antonfirsov reviewed Dec 4, 2021

View reviewed changes

kunalspathak added 15 commits August 16, 2022 23:26

Initial version of ImageSharp

6f9a958

Remove benchmark for Colorful

5daf1bf

Remove ImageMagick benchmarks

0ac04a7

Remove some other benchmarks

69d4abb

Copy TestUtilities

9191f47

Remove System.Drawing

3e1b24c

Remove unneeded benchmarks:

2525791

Delete more benchmarks and non essential files with working model

5307234

Remove Config files

664ab39

Fix categories

0c998b9

Delete SIMD based divide/multiply for now:

873cb1f

Add images

2be5aa3

Delete one more benchmark

6f6ab6e

Delete some webp images

87d1fb3

Trimmed more benchmarks based on review feedback

8ab68c4

kunalspathak marked this pull request as ready for review August 27, 2022 03:29

danmoseley added the adds new benchmarks label Aug 30, 2022

antonfirsov reviewed Sep 7, 2022

View reviewed changes

Upgrade Imagesharp to 2.1.3

609f4a2

kunalspathak mentioned this pull request Oct 20, 2022

Any consideration of having benchmarks? bepu/bepuphysics2#239

Open

kunalspathak added 3 commits October 28, 2022 07:06

Merge remote-tracking branch 'origin/main' into ImageSharp

c016e0f

review feedback

210841e

Update BDN and .NET to 7.0

dae25ad

Pass default CancellationToken

e2a8ec8

antonfirsov approved these changes Nov 3, 2022

View reviewed changes

Remove Resize_Bicubic_Compare_Rgba32_Rgb24

9732487

Fix lint markdown error

8dc3758

caaavik-msft mentioned this pull request Nov 4, 2022

Set up pipeline jobs for the Bepuphysics and ImageSharp benchmarks #2688

Closed

caaavik-msft approved these changes Nov 4, 2022

View reviewed changes

kunalspathak merged commit 7a7ac6f into dotnet:main Nov 4, 2022

kunalspathak deleted the ImageSharp branch November 4, 2022 05:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ImageSharp benchmark #2149

Add ImageSharp benchmark #2149

kunalspathak commented Nov 20, 2021

JimBobSquarePants commented Nov 22, 2021

danmoseley commented Nov 22, 2021

kunalspathak commented Nov 23, 2021

antonfirsov left a comment

kunalspathak commented Dec 2, 2021

kunalspathak commented Dec 2, 2021

antonfirsov left a comment

antonfirsov commented Dec 4, 2021 •

edited

Loading

danmoseley commented Feb 5, 2022

kunalspathak commented Feb 5, 2022

kunalspathak commented Aug 12, 2022

kunalspathak commented Aug 17, 2022

JimBobSquarePants commented Aug 17, 2022

kunalspathak commented Aug 17, 2022

kunalspathak commented Aug 22, 2022

antonfirsov commented Aug 22, 2022

kunalspathak commented Aug 22, 2022

kunalspathak commented Aug 27, 2022

kunalspathak commented Sep 2, 2022

JimBobSquarePants commented Sep 2, 2022

kunalspathak commented Sep 7, 2022

antonfirsov left a comment •

edited

Loading

antonfirsov Sep 7, 2022

antonfirsov Sep 7, 2022

antonfirsov Nov 3, 2022

kunalspathak commented Oct 28, 2022

antonfirsov commented Oct 28, 2022

kunalspathak commented Oct 28, 2022

kunalspathak commented Nov 1, 2022

JimBobSquarePants commented Nov 3, 2022

kunalspathak commented Nov 3, 2022

antonfirsov left a comment

antonfirsov Nov 3, 2022

kunalspathak commented Nov 4, 2022

Add ImageSharp benchmark #2149

Add ImageSharp benchmark #2149

Conversation

kunalspathak commented Nov 20, 2021

JimBobSquarePants commented Nov 22, 2021

danmoseley commented Nov 22, 2021

kunalspathak commented Nov 23, 2021

antonfirsov left a comment

Choose a reason for hiding this comment

kunalspathak commented Dec 2, 2021

kunalspathak commented Dec 2, 2021

antonfirsov left a comment

Choose a reason for hiding this comment

antonfirsov commented Dec 4, 2021 • edited Loading

danmoseley commented Feb 5, 2022

kunalspathak commented Feb 5, 2022

kunalspathak commented Aug 12, 2022

kunalspathak commented Aug 17, 2022

JimBobSquarePants commented Aug 17, 2022

kunalspathak commented Aug 17, 2022

kunalspathak commented Aug 22, 2022

antonfirsov commented Aug 22, 2022

kunalspathak commented Aug 22, 2022

kunalspathak commented Aug 27, 2022

kunalspathak commented Sep 2, 2022

JimBobSquarePants commented Sep 2, 2022

kunalspathak commented Sep 7, 2022

antonfirsov left a comment • edited Loading

Choose a reason for hiding this comment

antonfirsov Sep 7, 2022

Choose a reason for hiding this comment

antonfirsov Sep 7, 2022

Choose a reason for hiding this comment

antonfirsov Nov 3, 2022

Choose a reason for hiding this comment

kunalspathak commented Oct 28, 2022

antonfirsov commented Oct 28, 2022

kunalspathak commented Oct 28, 2022

kunalspathak commented Nov 1, 2022

JimBobSquarePants commented Nov 3, 2022

kunalspathak commented Nov 3, 2022

antonfirsov left a comment

Choose a reason for hiding this comment

antonfirsov Nov 3, 2022

Choose a reason for hiding this comment

kunalspathak commented Nov 4, 2022

antonfirsov commented Dec 4, 2021 •

edited

Loading

antonfirsov left a comment •

edited

Loading