Support for the display of various image/picture types (e.g., bitmaps)

[bmitc]

Is your feature request related to a problem? Please describe.
I would like to have images displayed in a .NET Interactive Notebook when working in Visual Studio Code with the .NET Interactive Notebooks extension. In particular, support for displaying SkiaSharp elements, such as images, bitmaps, etc. would be great.

Describe the solution you'd like
An example script might be:

#r "nuget: SkiaSharp"
open SkiaSharp
let bitmap = new SKBitmap(200, 200) // creates a default SKBitmap
// do some stuff to the bitmap, such as calling bitmap.SetPixel to manually set pixels
let image = SKImage.FromBitmap(bitmap) // creates an SKImage from a bitmap
let data = image.Encode() // encodes the image as a PNG

Then I'd like either of bitmap, image, or data left as the last element in the cell to have the underlying image displayed in the cell output, just as XPlot.Plotly does with its various charts and plots. The code above actually all works inside a notebook right now, and I'm able to save the image to a file. But I'd love the image to just directly appear in the cell's output.

For example:

If this is possible right now, then I am not sure how to get it to happen and would love any pointers if it is possible.

Describe alternatives you've considered
I considered trying to load various views that would load external to notebook, such as using SkiaSharp.Views.Forms, Xamarin.Forms, etc. However, that's not much different than just running things in an F# script (.fsx file). Having the image immediately displayed would be fantastic for all sorts of use cases.

Edit: I edited it because I submitted by accident while creating it.

[jonsequitur]

A custom implementation of this would be needed per .NET (base) type, so implementing this for SkiaSharp would require adding a .NET Interactive kernel extension to the SkiaSharp package.

You can see examples of how this is done here: https://github.com/dotnet/interactive/tree/main/samples/extensions

It might be worth supporting this out of the box for a number of common .NET types as well.

[bmitc]

Thanks for the pointer to the extensions!

And doing some more searching, I found out how XPlot.Plotly worked out of the box with notebooks in #582. I had searched for that before making this support request, as I was curious as to how that package did it, but had missed it.

I'll keep looking into the extensions, as I'd like to learn more about SkiaSharp anyway. As I understand from your comment and what I've read in the links you posted, SkiaSharp itself would take on an extension implementation addition to their package but not a package dependency on .NET Interactive. So to add this support, one would need to convince the SkiaSharp maintainers to add this extension implementation.

I think SkiaSharp would probably be one of the most useful additions for built-in .NET Interactive support (through extensions). Notebooks plus graphics serve as a perfect pairing, especially for documentation of and tutorials for the library and other fun stuff.

[mattleibow]

This is definitely something that I would want to include. My question is around the native things. How does this work? Each platform has a bunch of native files that do the rendering. Will the notebook load those as well?

[mattleibow]

I am just playing around with this using the ClockExtension and it seems that I have to put my extension in the "lib" folder of the NuGet? If I only include it in the interactive-extensions\dotnet directory, it is not loaded. If I include it in the lib folder, then it will get installed into apps?

Is is designed to be in a separate nuget package for extensions?

[jonsequitur]

Currently, yes. We are looking at another approach though where the extension could be a polyglot script in a .dib file.

[mattleibow]

I have just been playing around and I think this is all good.

I hacked a nuget together that has 3 files:

• lib\netstandard2.0\SkiaSharp.dll
• runtimes\win-x64\native\libSkiaSharp.dll
• interactive-extensions\dotnet\SkiaSharp.DotNet.Interactive.dll

It seems to need something in the lib folder to install since it uses the full dotnet system of packages? So if there was nothing in lib, then there was nothing to pull in. I could understand this might be an issue for extension-only packages, but we don't have this issue.

That just works! Great work team!

So basically, no extra work at all. No extra dependencies either. This is my full nuspec:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2012/06/nuspec.xsd">
  <metadata>
    <id>SkiaSharp.DotNet.Interactive</id>
    <version>0.0.1-preview.2021.5.29.15.40.17</version>
    <authors>SkiaSharp.DotNet.Interactive</authors>
    <description>Renders a clock in dotnet-interactive using SVG</description>
    <dependencies>
      <group targetFramework="net5.0" />
    </dependencies>
  </metadata>
  <files>
    <file src="C:\Users\matthew\.nuget\packages\skiasharp\2.80.2\lib\netstandard2.0\SkiaSharp.dll" target="lib\netstandard1.0\SkiaSharp.dll" />
    <file src="C:\Users\matthew\.nuget\packages\skiasharp\2.80.2\runtimes\win-x64\native\libSkiaSharp.dll" target="runtimes\win-x64\native\libSkiaSharp.dll" />
    <file src="C:\Users\matthew\.nuget\packages\skiasharp\2.80.2\runtimes\win-x86\native\libSkiaSharp.dll" target="runtimes\win-x86\native\libSkiaSharp.dll" />
    <file src="C:\Users\matthew\.nuget\packages\skiasharp\2.80.2\runtimes\osx\native\libSkiaSharp.dylib" target="runtimes\osx\native\libSkiaSharp.dylib" />
    <file src="C:\Projects\SkiaSharp_80\source\SkiaSharp.DotNet.Interactive\bin\Debug\net5.0\SkiaSharp.DotNet.Interactive.dll" target="interactive-extensions\dotnet\SkiaSharp.DotNet.Interactive.dll" />
  </files>
</package>

[jonsequitur]

You should be able to build a simpler package that just references the SkiaSharp package rather than directly including the DLLs.

[mattleibow]

I want to include the extension dlls in SkiaSharp (since I am the author) and don't want to have to install a special package.

[mattleibow]

One thing I did notice with the packages, the version in the sample ClockExtension causes a runtime exception with a missing method Display in the latest stable vscode.

[mattleibow]

How would one render images? Just convert/encode the bitmap to base64? Or is there some special way for images?

[AriaMoKr]

Just a simple sample using base64 as @mattleibow mentioned and PocketView:

`
using System;
using System.IO;
using System.Drawing;
using System.Drawing.Imaging;
using Microsoft.DotNet.Interactive.Formatting;

MemoryStream memStream;
Bitmap b = new Bitmap(50, 50);
Graphics g = Graphics.FromImage(b);

void d()
{
memStream = new MemoryStream();
b.Save(memStream, ImageFormat.Png);
string base64str = Convert.ToBase64String(memStream.ToArray());
display(PocketViewTags.img[src: "data:image/png;base64," + base64str]);
}

g.Clear(Color.White);
g.DrawLine(Pens.Black, 0, 0, b.Width, b.Height);
d();

g.Clear(Color.Black);
g.DrawRectangle(Pens.Blue, 5, 5, b.Width-10, b.Height-10);
d();
`

should give a results like:

[mattleibow]

In the latest previews of SkiaSharp I added some elements, but I also had to use base64.

Would be cool to be able to do this automatically. And if it supports images, it might be nice for workbooks to have some basic info like size and stuff.

[colombod]

Are you already creating a .NET Interactive extension that formats your types? Have a look at this extension for ImageSharp

[AriaMoKr]

Just curious, would sending a MemoryStream, a Bitmap, or a Graphics object to the display function be recommended?
I'm guessing not a Graphics object. Maybe MemoryStream in this case sense there's already support to display a PNG or would Bitmap be less to support?

[colombod]

Just curious, would sending a MemoryStream, a Bitmap, or a Graphics object to the display function be recommended?
I'm guessing not a Graphics object. Maybe MemoryStream in this case sense there's already support to display a PNG or would Bitmap be less to support?

Have you seen this sample your comment about the Graphic object in interesting

[AriaMoKr]

Just curious, would sending a MemoryStream, a Bitmap, or a Graphics object to the display function be recommended?
I'm guessing not a Graphics object. Maybe MemoryStream in this case sense there's already support to display a PNG or would Bitmap be less to support?

Have you seen this sample your comment about the Graphic object in interesting

I had not seen the sample, thanks for sharing. I haven't dug into Graphic but just figured the other two objects I had used would be simpler to convert.

[mattleibow]

My current implementation lives here: https://github.com/mono/SkiaSharp/tree/main/source/SkiaSharp.DotNet.Interactive

[colombod]

My current implementation lives here: https://github.com/mono/SkiaSharp/tree/main/source/SkiaSharp.DotNet.Interactive

Looks good, to make sure to have this part in your csproj

  <ItemGroup>
    <None Include="$(OutputPath)/SkiaSharp.DotNet.Interactive.dll" Pack="true" PackagePath="interactive-extensions/dotnet" />
  </ItemGroup>

[rdthree]

Hi - I'm terribly new to this whole .NET Interactive notebooks thing. I could not figure out how to get @mattleibow 's extension installed. However, digging around the docs and tooling around, it looks like SKSurface Class has a .Snapshot() method that works in the notebook without any added extensions.

• https://docs.microsoft.com/en-us/dotnet/api/skiasharp.sksurface?view=skiasharp-2.80.2

Also, the display command doesn't seem to work anymore in the notebook. The notebooks seem to have their own .Display() method. I'm guessing because all this stuff is so new its hard to find documentation for n00bs like me?

#r "nuget: SkiaSharp"
using SkiaSharp;

SKImageInfo info = new SKImageInfo(256, 256);
SKSurface surface = SKSurface.Create(info);
SKCanvas canvas = surface.Canvas;

canvas.Clear(SKColors.White);

// configure our brush
SKPaint redBrush = new SKPaint {
    Color = new SKColor(0xff, 0, 0),
    IsStroke = true
};
SKPaint blueBrush = new SKPaint {
    Color = new SKColor(0, 0, 0xff),
    IsStroke = true
};

for (int i = 0; i < 64; i += 8) {
    SKRect rect = new SKRect(i, i, 256 - i - 1, 256 - i - 1);
    canvas.DrawRect(rect, (i % 16 == 0) ? redBrush : blueBrush);
}

surface.Snapshot().Display();

[bmitc]

@rdthree SkiaSharp is a general API for drawing. For .NET Interactive notebooks, one can implement some custom formatters so that datatypes get specially displayed in the notebook (such as Plotly.NET). To get the interactive behavior, you sometimes need to reference another Nuget package (such as how Plotly.NET does it). I think @mattleibow's plan was to include the SkiaSharp extension in the already existing Nuget package, so you wouldn't need to separately reference an extension package. He added some basic support (see here and here), but I'm not sure if it got released in a Nuget release and/or if it's still current. I believe he did it as a quick trial implementation where it wasn't intended to be feature complete.

I didn't know surface.Snapshot().Display() would work like that, so maybe that's the extension at work (?) or some new features of .NET Interactive (?). Someone else would know. I just created this feature suggestion. :)

[bmitc]

@jonsequitur @colombod

I long ago offered to take a look at implementing this, but that got curtailed with other stuff. However, I actually just this weekend started looking back into this and would like to take it on as a project to fully implement the SkiaSharp formatter.

So, I started collecting resources. Are these the most up to date extension/formatter how-tos?

1. https://github.com/dotnet/interactive/tree/main/samples/extensions
2. https://github.com/dotnet/interactive/blob/main/docs/extending-dotnet-interactive.md

For other examples and references, I've collated:

1. The initial support added to SkiaSharp: Add .NET Interactive Support mono/SkiaSharp#1710
2. Formatters should support multiple MIME types as a standard behavior #1374
3. System.Drawing extension
4. ImageSharp extension
5. Plotly.NET extension, which uses F#

Any other advice? I'm going to start going through everything and start trying something out. My plan is to do it in F#.

[jonsequitur]

@bmitc, those extensions docs are still current. One addition to be aware of is script-based extensions. For smaller amounts of code, this might be more sustainable because it's less likely to result in dependency conflicts. This story still has some rough edges and we'd appreciate any suggestions.

And it looks like the SkiaSharp package does include the extension. When I ran @rdthree's code I saw this:

[bmitc]

@jonsequitur Thanks for the confirmation and the mention of the script-based extensions! Those seem like a nice middle ground where a user only has to reference a single NuGet package (at least in the case of SkiaSharp) and gets the .NET Interactive goodies without non-.NET Interactive users taking on additional dependencies (in the case of SkiaSharp not wanting a separate NuGet package for the interactive bits). I can check with @mattleibow once I get something working and understood from my side.

A follow-up question is, and maybe this is a dumb one: does the script-based extension allow me to simply reference SkiaSharp in a notebook and then develop and test the extension in the notebook? Or maybe it's the case that that was already possible, and this feature just adds some automatic loading of the file as mentioned when referencing a given NuGet package?

[jonsequitur]

does the script-based extension allow me to simply reference SkiaSharp in a notebook and then develop and test the extension in the notebook?

Yes. Anything you can run in the notebook can be put into the script in exactly the same way. You don't have to translate notebook code into project code to go from experimenting in the notebook to distributing via a package. The script will run against whatever dependencies are loaded, including the main assembly in the NuGet package. It's intended to be an alternative to the assembly-based extension approach if your code is fairly simple but it also makes automated testing and refactoring more difficult. I don't think we've found the right developer experience here yet so we'd appreciate your thoughts.

[nullptrerror]

PocketViewTags.img[src: "data:image/png;base64," + base64str]

This is so nice.

[luiscib3r]

void DisplayImage(byte[] image, string format = "jpeg", double? width = null, double? height = null) {
    var base64 = Convert.ToBase64String(image);
    var imageHtml = $"<img src='data:image/{format};base64,{base64}" +
        (width.HasValue ? $" width='{width.Value}px'" : "") +
        (height.HasValue ? $" height='{height.Value}px'" : "") +
        "'/>";
    imageHtml.DisplayAs("text/html");
}