Image Processing

Note: You are currently viewing the v0.7 Alpha examples. If you are not cloning the latest code from this repository then you may wish to look at the v0.6 examples instead.

Revisions

• v0.7 Alpha
• v0.6 (Current)
• v0.5

Contents

1. Sharpen
2. Edge Detection
3. Line Detection
4. Gaussian Blur
5. Box Blur
6. Strip Bayer metadata
7. Custom convolution
8. Advanced: convolution cell counts

Most of the image processing techniques available currently in MMALSharp are based on Matrix Convolutions which can be found on Wikipedia.

Sharpen

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new SharpenProcessor());
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Edge Detection

There are 3 modes available which provide different strengths of Edge Detection: EDStrength.Low, EDStrength.Medium and EDStrength.High.

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new EdgeDetection(EDStrength.High));
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Line Detection

Line detection detects edges with specific orientations:

• Horizontal
• Vertical
• Diagonal Down (sloping down left-to-right)
• Diagonal Up (sloping up left-to-right)
• Sobel Horizontal
• Sobel Vertical

Sobel filters apply a softening effect to reduce sensitivity to noise.

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new LineDetection(LineDetectionType.Vertical));
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Gaussian Blur

As per the Matrices available on the Wikipedia page, MMALSharp provides functionality for a 3x3 or 5x5 Gaussian Blur matrix convolution.

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new GaussianProcessor(GaussianMatrix.Matrix3x3));
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Box Blur

Box blur is a different blurring effect best-suited to natural scenes (few edges and hard corners).

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new BoxBlur());
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Strip Bayer Metadata

Use the following example to get raw access to the Bayer metadata produced with a JPEG frame. Ensure you pass in the correct camera version to the BayerMetaProcessor constructor, either CameraVersion.OV5647 or CameraVersion.IMX219.

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder(true))
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new BayerMetaProcessor(CameraVersion.OV5647));
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Custom convolution

By using the CustomConvolutionProcessor class, you can create your own custom kernel convolutions and have them applied to an image captured by MMALSharp. Simply pass in the kernel array, kernel width and height and the image will be modified accordingly.

private const int KernelWidth = 3;
private const int KernelHeight = 3;
private double[,] TopSobelKernel = new double[KernelWidth, KernelHeight]
{
    { 1, 2, 1 },
    { 0, 0, 0 },
    { -1, -2, -1 }
};

public async Task TakePictureManual()
{                        
    MMALCamera cam = MMALCamera.Instance;

    using (var imgCaptureHandler = new ImageStreamCaptureHandler("/home/pi/images/", "jpg"))
    using (var imgEncoder = new MMALImageEncoder())
    using (var nullSink = new MMALNullSinkComponent())
    {
        cam.ConfigureCameraSettings();
        
        var portConfig = new MMALPortConfig(MMALEncoding.JPEG, MMALEncoding.RGB24, quality: 90);

        // Create our component pipeline.         
        imgEncoder.ConfigureOutputPort(portConfig, imgCaptureHandler);
                
        cam.Camera.StillPort.ConnectTo(imgEncoder);                    
        cam.Camera.PreviewPort.ConnectTo(nullSink);
        
        imgCaptureHandler.Manipulate(context =>
        {
            context.Apply(new CustomConvolutionProcessor(this.TopSobelKernel, KernelWidth, KernelHeight));
        }, ImageFormat.Jpeg);
        
        // Camera warm up time
        await Task.Delay(2000);        
        await cam.ProcessAsync(cam.Camera.StillPort);
    }

    // Only call when you no longer require the camera, i.e. on app shutdown.
    cam.Cleanup();
}

Advanced: convolution cell counts

Each convolution class constructor has an override which accepts two additional integer arguments: horizontalCellCount and verticalCellCount. This lets you fine-tune convolution processing for performance-sensitive uses.

A parallel processing algorithm applies the convolutions to the images. It does this by dividing the image into an array of rectangles. The library has preset values for all of the V1, V2, and HQ camera resolutions, but you can supply custom values using these alternate constructors. You should choose values that divide evenly into the horizontal and vertical resolutions. On a Raspberry Pi 4B, a total cell count in the range of 700 to 1100 cells provides the best throughput. For example, the 1920 x 1080 resolution has a default horizontal and vertical cell count of 30, yielding 900 total cells which are 64 x 36 pixels each.

For maximum performance, you can apply convolutions to raw image data. In these cases you must account for buffer padding -- the raw buffer's horizontal size will be aligned to the next highest 32-byte value, and the vertical size will be aligned to the next highest 16-byte value. The raw buffer for a 1920 x 1080 image is sized at 1920 x 1088, so the default cell counts become 30 x 32, for a total of 960 cells at 64 x 34 pixels per cell.