F score over boundaries

[bhack]

🚀 Feature

An F score over the boundaries.
It will requires boundaries extraction on multiclass tensors.

Motivation

It is used in many Video dataset and their related challenges (Davis, Yotube VOS, etc.).

Pitch

Generally we have opencv/script/numpy implementations but it could be very useful to have a metrics directly to plug in the eval step.

Alternatives

Implementing BoundaryIOU
https://bowenc0221.github.io/boundary-iou/

Additional context

F-boundary Reference implementation:
https://github.com/davisvideochallenge/davis2017-evaluation/blob/master/davis2017/metrics.py

[github-actions]

Hi! thanks for your contribution!, great first issue!

[SkafteNicki]

Hi @bhack, thanks for proposing this metric to be a part of torchmetrics.
One question: what would typical examples for such a metric look like?

[bhack]

Check Section 4.1 in
https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Perazzi_A_Benchmark_Dataset_CVPR_2016_paper.pdf

You will find also a comparative analysis where the alternative BoundaryIOU was proposed at:
https://arxiv.org/pdf/2103.16562.pdf

But F is still the popular one for boundaries measurement in dedicated conference challenges/workshop.

I suppose that probably we could reuse the F1Score multiclass metric already available in the library but it needs to be integrated by a boundary extraction function.

Let me know if you need any other info.

[bhack]

@vkosolapov Has a torchmetric for BoundaryIOU but still relaying on OpenCV for extracting the boundarties:

https://github.com/vkosolapov/anchor_free/blob/master/anchor_free/model/metric.py#L8-L74

[bhack]

If it could be useful this is mask_to_boundary pytorch equivalent implementation to the reference BoundaryIOU implementation for extracting the mask boundaries.

Refactored it could be probably quite compatible/reused also for the F metric.

def mask_to_boundary_pytorch(mask, dilation_ratio=0.02):
    # convert mask to float tensor
    mask = mask.float()

    # calculate dilation ratio based on image diagonal
    h, w = mask.shape
    img_diag = math.sqrt(h ** 2 + w ** 2)
    dilation = int(round(dilation_ratio * img_diag))

    # create structuring element for morphological operations
    selem_size = dilation * 2 + 1
    selem = torch.ones(selem_size, selem_size, device=mask.device)

    # erode the mask using the structuring element
    eroded_mask = F.conv2d(mask.unsqueeze(0).unsqueeze(0), selem.unsqueeze(0).unsqueeze(0), padding=dilation)
    eroded_mask = eroded_mask.squeeze(0).squeeze(0)
    eroded_mask = (eroded_mask == (selem.sum()))  # make the output binary

    # subtract the eroded mask from the original mask to get the contour
    contour = mask.byte() - (eroded_mask > 0).byte()

    return contour

[bhack]

@Borda Can the mask to boundary util be contributed somewhere in the meantime? I don't see a generic util file in the image folder.

[DimitrisMantas]

This is a very useful metric for semantic segmentation since it correlates nicely with human perception of IoU.

Any news on whether it will be added?