Optional Metrics

[EricZimmermann]

Hello!

As mentioned in MONAI discussion here: Project-MONAI/MONAI#3392

Ignite metrics call .update on iteration completed event. In the case where you have a metric that has a NoneType value, is there a recommended way of processing things?

Example:
Given data coming from N sources, I would like to compute per source metrics. For each iteration, I sample M independent items where M < N. This means that M metrics will have a value, and (N-M) will be empty (or NoneType) because no sample to access via output transform for that metric. For the empty values, the engine still calls metric.update but has nothing to access. In the case where it is filled with zeros to try and mitigate this issue, any accumulated metric would also increment leading to invalid summary stats.

Solution:
Wrap metrics to handle NoneTypes (see code snippet below)

import types

def optional_metric(metric: Metric):

    '''
        Decorate a ignite metric to be optional
        Dict must return valid keys with ignored value as NoneType
        Use case: multi-source statistics
    '''

    base = metric.update                                        # reference to update
    def optional_update(self, outputs):                         # check nonetypes
        if isinstance(outputs, Sequence):
            isCallable =  all(map(lambda o: o != None, outputs))
        else: 
            isCallable = (outputs is not None)
        if isCallable:
            base(outputs)
    
    metric.update = types.MethodType(optional_update, metric)  # decoration
    return metric

I also have a class to format things but did not include it. As mentioned, is there a better or more efficient way to do this currently?

Best,
Eric, @JustinSzeto, @kirillv-cim

[vfdev-5]

@EricZimmermann I think we have already though about skipping metric's update if None is passed however there was a blocker from the implementation of RunningAverage.
However, I could find this example : #1310 (comment) in our questions which may help filtering the updates...

I'll provide a prototype example for your use-case here a bit later.

[vfdev-5]

Here is a prototype code using BatchFiltered metric usage for your use-case (at least how I understood it):

import torch
from functools import partial

from ignite.metrics import Accuracy
from ignite.engine import Engine, Events

from ignite.metrics.metric import BatchFiltered

n_iters = 20
batch_size = 2
n_sources = 5  # N
n_classes = 10
n_sampled_sources = 3  # M  

logits_list = torch.rand(n_iters, n_sources, batch_size, n_classes)
labels_list = torch.ones(n_iters, n_sources, batch_size)


def validation_step(e, _):
    i = e.state.iteration - 1
    logits_to_sample = logits_list[i]
    labels_to_sample = labels_list[i]
    
    # THIS IS HOW I SAMPLE FROM M SAMPLES FROM N SOURCES
    sampled_source_idx = torch.randint(0, n_sources, size=(n_sampled_sources, ))
    print("sampled_source_idx: ", sampled_source_idx.tolist())
    
    return [
        (logits_to_sample[i], labels_to_sample[i]) if i in sampled_source_idx else None
        for i in range(n_sources)
    ]

def filterout_nones(e, ev, src):
    if e.state.output[src] is None:
        print(f"Skip {src} output")
        return False
    return True


evaluator = Engine(validation_step)

@evaluator.on(Events.ITERATION_COMPLETED)
def print_output():
    print(evaluator.state.iteration, " - ", ["Entry" if o is not None else "None" for o in evaluator.state.output])

    
def get_output(output, src):
    return output[src]

metrics = {}
for i in range(n_sources):
    metrics[i] = Accuracy(output_transform=partial(get_output, src=i))  # can't use lambda in for-loop
    # Here we are defining BatchFiltered with a custom event filter that is aware of the source index
    metrics[i].attach(evaluator, f"acc_{i}", usage=BatchFiltered(event_filter=partial(filterout_nones, src=i)))

state = evaluator.run(range(20))
print(state.metrics)

sampled_source_idx:  [3, 1, 4]
1  -  ['None', 'Entry', 'None', 'Entry', 'Entry']
Skip 0 output
Skip 2 output
sampled_source_idx:  [1, 3, 1]
2  -  ['None', 'Entry', 'None', 'Entry', 'None']
Skip 0 output
Skip 2 output
Skip 4 output
sampled_source_idx:  [1, 0, 4]
3  -  ['Entry', 'Entry', 'None', 'None', 'Entry']
Skip 2 output
Skip 3 output
...
sampled_source_idx:  [4, 2, 3]
19  -  ['None', 'None', 'Entry', 'Entry', 'Entry']
Skip 0 output
Skip 1 output
sampled_source_idx:  [0, 3, 1]
20  -  ['Entry', 'Entry', 'None', 'Entry', 'None']
Skip 2 output
Skip 4 output
{'acc_0': 0.0625, 'acc_1': 0.07142857142857142, 'acc_2': 0.15, 'acc_3': 0.25, 'acc_4': 0.2}

Hope this helps !

[EricZimmermann]

@vfdev-5 Thank you so much for taking the time to write this out!
I will take a better look at this with my group and discuss! What was the issue with RunningAverage? Is there a thread for this?

[vfdev-5]

Sure, let me know if this answers your use-case.

What was the issue with RunningAverage?

We wanted to generalize Metric behaviour such that if enigne.state.output is None, then nothing is updated automatically. RunningAverage implementation is very particular and the problem was how to skip also compute , #1065 (comment)