[ML Text Recognition] Drawing shapes for slightly rotated images (On captured image)

[Spiderixius]

Hello!
Background
I am using Firebase text recognition to recognise text in images. However I have the following problem:

As you can see in the image if the target image is slightly rotated then the frames are being overlapped. What I would like to achieve is for the blue rectangles to follow the text rather than having a horizontal shape. Currently I do drawing in the following manner, inside of my processText(...) method:

// Paragraphs
for block in text.blocks {
    // Lines
    for line in block.lines {
        drawFrame(line.frame, transform: transform)
    }

The drawFrame(...) method does the following:

private func drawFrame(_ frame: CGRect, transform: CGAffineTransform) {
    let transformedRect = frame.applying(transform)
    UIUtilities.addRectangle(transformedRect, to: self.annotationOverlayView)
}

So, once the line element is transformed I do the drawing:

public static func addRectangle(_ rectangle: CGRect, to view: UIView){
    let rectangleView = UIView(frame: rectangle)
    rectangleView.layer.borderColor = UIColor(hex: Constants.colorDarkBlue).cgColor
    rectangleView.layer.borderWidth = Constants.lineWidth
    rectangleView.bounds = rectangleView.frame.insetBy(dx:-3, dy: -3)
    view.addSubview(rectangleView)
}

What did I try?
I tried looking at the addShape(...) method in this repository: https://github.com/firebase/quickstart-ios/blob/5b752734233d4be625ece2747385d4974f2fc07f/mlvision/MLVisionExample/UIUtilities.swift#L52
I am 100% confident that `addShape(...) is the method to use but I I can see in the example during the live run example it is capable of adjusting the shape of the highlight regardless of how much I rotate the image.

So I tried to do the following:
This is inside my processResults(...) method. The captureImageView is the view in which I put the captured image on and then do processing on it.

// Paragraphs
for block in text.blocks {
    // Lines
    for line in block.lines {
        let points = self.convertedPoints(from: line.cornerPoints, width: capturedImageView.bounds.width, height: capturedImageView.bounds.height)
        print("Points: \(points)")
        UIUtilities.addShape(
             withPoints: points,
             to: self.annotationOverlayView,
             color: UIColor.orange)

Similar to the Repository I link above, I copied over the addShape(..) into my class. I also coppied convertedPoints and normalizedPoint
You can see in my convertedPoints(...) I am not actually using the normalizedPoint constant (as I am not sure what do there).

private func convertedPoints(
    from points: [NSValue]?,
    width: CGFloat,
    height: CGFloat
    ) -> [NSValue]? {
    return points?.map {
        let cgPointValue = $0.cgPointValue
        let normalizedPoint = CGPoint(x: cgPointValue.x / width, y: cgPointValue.y / height)
        //let cgPoint = previewLayer.layerPointConverted(fromCaptureDevicePoint: normalizedPoint)
        let cgPoint = capturedImageView.frame.origin
        let value = NSValue(cgPoint: cgPoint)
        return value
    }
}

And this is my normalizePoint(...) method

private func normalizedPoint(
    fromVisionPoint point: VisionPoint,
    width: CGFloat,
    height: CGFloat
    ) -> CGPoint {
    let cgPoint = CGPoint(x: CGFloat(point.x.floatValue), y: CGFloat(point.y.floatValue))
    var normalizedPoint = CGPoint(x: cgPoint.x / width, y: cgPoint.y / height)
    normalizedPoint = capturedImageView.frame.origin
    return normalizedPoint
}

To Lazy to Read
I would like to adjust the shape of the highlight to adapt to slightly rotated images. Similar to the example mlvision application (With live AVFoundation camera run) in this repository.

[morganchen12]

The mlvision sample doesn't do any slanted highlighting as far as I know. The highlighting logic is here. You may have better luck asking how to draw slanted rectangles on StackOverflow.

[Spiderixius]

@morganchen12 It does! The following image illustrates that https://imgur.com/vTd8LqE This is using the MlVision example from this repository. At the top right there is a camera icon. The difference is, it does such detection for AVPreviewLayer, so the image is constantly being buffered. But what I would like to do is to only use it with an image I have captured.

Additionally, I have already asked my question on StackOverflow as well: https://stackoverflow.com/questions/57197271/draw-appropriate-word-highlight-shapes-for-slightly-rotated-images-with-firebase

[Spiderixius]

Any update on this, or should I just close the issue?