Add support for custom evaluation metrics

[brimoor]

Setup

1. Make sure you are running the custom-metrics branch of your fiftyone install
2. If you haven't already, install fiftyone-plugins in editable mode:

git clone https://github.com/voxel51/fiftyone-plugins
cd fiftyone-plugins
git checkout --track origin/custom-metrics

ln -s "$(pwd)" "$(fiftyone config plugins_dir)/fiftyone-plugins"

Example usage

example_metric

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart")

#
# Option 1: App usage
#
# Launch the `evaluate_model` operator from the Operator Browser
# Scroll to the bottom of the modal to choose/configure custom metrics to apply
#
# Use the `delete_evaluation` operator to cleanup an evaluation
#

session = fo.launch_app(dataset)

#
# Option 2: SDK usage
#

# A custom metric to apply
metric = "@voxel51/metric-examples/example_metric"
kwargs = dict(value="spam")

results = dataset.evaluate_detections(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
    custom_metrics={metric: kwargs},
)

print(dataset.count_values("eval_example_metric"))
results.print_metrics()

dataset.delete_evaluation("eval")

assert not dataset.has_field("eval_example_metric")

absolute_error and squared_error

Image dataset

import random
import numpy as np

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("cifar10", split="test")
dataset.delete_sample_field("ground_truth")

for idx, sample in enumerate(dataset.iter_samples(autosave=True, progress=True), 1):
    ytrue = random.random() * idx
    ypred = ytrue + np.random.randn() * np.sqrt(ytrue)
    confidence = random.random()
    sample["ground_truth"] = fo.Regression(value=ytrue)
    sample["predictions"] = fo.Regression(value=ypred, confidence=confidence)

results = dataset.evaluate_regressions(
    "predictions",
    gt_field="ground_truth",
    eval_key="eval",
    custom_metrics=[
        "@voxel51/metric-examples/absolute_error",
        "@voxel51/metric-examples/squared_error",
    ],
)

print(dataset.bounds("eval_absolute_error"))
print(dataset.bounds("eval_squared_error"))

results.print_metrics()

dataset.delete_evaluation("eval")

assert not dataset.has_field("eval_absolute_error")
assert not dataset.has_field("eval_squared_error")

Video dataset

import random
import numpy as np

import fiftyone as fo
import fiftyone.zoo as foz

dataset = foz.load_zoo_dataset("quickstart-video")
dataset.delete_frame_field("detections")

for sample in dataset.iter_samples(autosave=True, progress=True):
    for frame_number, frame in sample.frames.items():
        ytrue = random.random() * frame_number
        ypred = ytrue + np.random.randn() * np.sqrt(ytrue)
        confidence = random.random()
        frame["ground_truth"] = fo.Regression(value=ytrue)
        frame["predictions"] = fo.Regression(
            value=ypred, confidence=confidence
        )

results = dataset.evaluate_regressions(
    "frames.predictions",
    gt_field="frames.ground_truth",
    eval_key="eval",
    custom_metrics=[
        "@voxel51/metric-examples/absolute_error",
        "@voxel51/metric-examples/squared_error",
    ],
)

print(dataset.mean("eval_absolute_error"))
print(dataset.mean("eval_squared_error"))
print(dataset.bounds("frames.eval_absolute_error"))
print(dataset.bounds("frames.eval_squared_error"))

results.print_metrics()

dataset.delete_evaluation("eval")

assert not dataset.has_field("eval_absolute_error")
assert not dataset.has_field("eval_squared_error")
assert not dataset.has_field("frames.eval_absolute_error")
assert not dataset.has_field("frames.eval_squared_error")

[ritch]

This really should just be baked in... having the contract defined here is not maintainable. Something like

foo.list_operators_with_metadata(type="operator", eval_type=eval_type, method=method)

[ritch]

Although the logic here should be colocated with the custom metric implementation, so that part is fine for now.

[brimoor]

yes I agree, was just getting something working to start.

More generally, we should have a think about the concept of "operator classes" like EvaluationMetric(Operator) introduced here and how we want that to work holistically. For example, should the concept of tags be in fiftyone.yml rather than OperatorConfig so that we can access information about the type of operators without having to instantiate them?

[manushreegangwar]

I would prefer to keep an EvaluationMetric operator agnostic to eval_type and method here since certain metrics may not fall under any of the eval methods or may be applicable to more than one eval method.

To start with, the users should be free to choose whichever custom metrics they want and if the metric is not applicable to the eval method under consideration, it fails gracefully (which is already the case).

[brimoor]

The implementation here does allow for eval_type and method to be optional.

If the EvaluationMetric defines them, then the model evaluation operator will only show the metric as an option in the dropdown if the user is executing the matching type/method evaluation.

If the EvaluationMetric does not define them, then the metric will always be included as an available metric in the dropdown.

[manushreegangwar]

Your design works fine for what we currently have as evaluation methods.

I was thinking about evaluation methods that don't fall under the ones currently available. For example, if I am evaluating a neural rendering model, I may want to reuse SquaredErrorMetric which is of type "regression" and also "SSIM" which is not of type "regression".

[brimoor]

Ah so you want a way to declare that an evaluation metric should be available to multiple evaluation types, ie type=["regression", "neural-rendering"]?

That's certainly possible.

[brimoor]

An alternative model would be that authors of metrics encode semantics of the metric is tags:

class MyEvaluationMetric(EvaluationMetric):
    @property
    def config(self):
        return foo.OperatorConfig(
            ...
            tags=["metric", "regression"],
        )

and then when new evaluation protocols are added, they have a way to declare that they support any custom metrics with certain tags(s):

def evaluate_neural_rendering(...):
    supported_metric_tags = ["regression", ...]

[manushreegangwar]

The alternative model looks good. I am assuming we will have a lot more custom metrics than evaluation methods so my preference is to let the evaluation method choose the metrics it supports.

[ritch]

A few comments for the final implementation:

1. We should require EvaluationMetric and isinstance(operator, EvaluationMetric)
2. We should define the EvaluationMetric specific configuration options in a EvaluationMetricConfig class
3. Instead of "tags" name the param metric_protocols to keep it specific and simple

[brimoor]

@ritch okay I've implemented what you recommended here in these commits:

• fiftyone-plugins: 727d204
• fiftyone: voxel51/fiftyone@e98cea0

[manushreegangwar]

What's unlisted for?

[brimoor]

Operators that are unlisted=True will not appear in the Operator Browser.

All metrics should be marked as unlisted because they aren't intended for direct execution like that (which requires implementing the resolve_input() and execute() methods).

[brimoor]

@ritch and I discussed wanting to have a way to automatically mark all EvaluationMetric operators as unlisted, but that's not implemented yet.