[Feature] Add EvalHook which will be used in downstream projects

[dreamerlin]

Overview

This PR adds mmcv/runner/hooks/eval.py file and EvalHook and DistEvalHook to MMCV together with related unittest parts. Since those two hooks have been widely used in many MM repos, this PR implements and unifies common functions used in the evaluation part.

Design

Model Evaluation performed in OpenMMLab uses EvalHook and DistEvalHook during training.

In detail, the those hooks are commonly registered in apis/train.py by runner.register_hook(eval_hook(val_dataloader, **eval_cfg)). Users can initialize their models with following steps:

1. Define validation dataset by assigning cfg.data.val, building val_datasetloader
2. Register EvalHook using runner.register_hook.

Once registering it, runner will periodically call the evaluate function to perform evaluation in a fixed mode (by epoch or by iteration, etc.). The high-level workflow of EvalHook in OpenMMLab is:

register EvalHook -> define evaluation setting (start, interval, by_epoch, save_best, comparison rules, etc.) -> perform evaluation after_train_iter or after_train_epoch -> (save the best checkpoint) -> loop back to after_train_xxx

APIs

EvalHook is the base module, and DistEvalHook is a child class to it. They both inherit Hook. Here is the APIs explanation for EvalHook.

Initialization

• dataloader: The PyTorch dataloader for evaluation. It can be built with build_dataloader with provided dataloader setting.

• start: Evaluation starting epoch. It enables evaluation before the training starts if start <= the resuming epoch. If set to None, whether to evaluate is merely decided by interval.

• interval: Evaluation interval.

• by_epoch: Determine whether to perform evaluation by epoch or by iteration. If set to True, it will perform by epoch. Otherwise, by iteration.

• save_best: If a metric is specified, it would measure the best checkpoint during evaluation. The information about best checkpoint would be save in runner.meta['hook_msgs']. Options are the evaluation metrics to the validation dataset. e.g., bbox_mAP, segm_mAP for bbox detection and instance segmentation. AR@100 for proposal recall. If save_best is auto, the first key of the returned OrderedDict result will be used. The interval of CheckpointHook should be divisible by that of EvalHook.

• rule : Comparison rule for best score. If set to None, it will infer a reasonable rule. Keys such as acc, top, etc. will be inferred by greater rule. Keys contain loss will be inferred by less rule. Options are greater, less, None.

  But Note: Since the rule for downstream repos are different, it may need to overwrite the self.greater_keys and self.less_keys

• eval_kwargs: Key word arguments for dataset evaluate function (def evaluate), which will be fed into the evaluate function of the dataset.

• Other hardcoded inner variables: rule_map, init_value_map, greater_keys and less_keys.

  Note that: Since the rule for downstream repos are different, it may need to overwrite these variable due to the specific task.

  • rule_map: A dict containing comparison function, default as {'greater': lambda x, y: x > y, 'less': lambda x, y: x < y}.
  • init_value_map: The initialized value for comparison, default as {'greater': -inf, 'less': inf}.
  • greater_keys : A list containing some rule keys applied for greater function, which means in these rules EvalHook regards greater number as the better one. Note that: If a string is one of the rules' substring, it will be applied to greater function. e.g., for acc: top1_acc, top5_acc, mean_acc are all applied to greater rule.
  • less_keys: Similar to greater_keys but it is for less rules. e.g., for loss: bce_loss, bmn_loss are all applied to less rule.

before_run

This part is to initialize runner.meta.hook_msgs, if users determine to save the best checkpoint.

before_train_xxx

For before_train_iter and before_train_epoch part. It is mainly to determine whether it is the right time to perform evaluation by examining:

• Whether it is the correct time to perform evaluation according to by_epoch
• Whether it achieves the start to perform evaluation

And use self.initial_flag to indicate whether the EvalHook gets into the normal evaluation loop. After getting into the normal evaluation loop, before_train_xxx will be skipped.

after_train_xxx

For after_train_iter and after_train_epoch part. It is mainly to inference the model and do evaluation, as well as save the best checkpoint if save_best is specified. In detail, it will call _do_evaluate() and _save_best().

• _do_evaluate(): inference the model by calling single_gpu_test(), do evaluation and call _save_best()
• _save_best(): compare the score according to the rule, write info into runner.meta['hook_msgs'] and save the best checkpoint in the work_dir

For DistEvalHook, besides the variable and function mentioned above, it inferences model by calling multi_gpu_test() and assigns tmpdir, gpu_collect for multi_gpu_test()

Usages

Users adopt EvalHook by typing in --validate for training command, it will call EvalHook in training.

Config

To use EvalHook with a specific setting, users need to modify the evaluation variable in config files. like this:

evaluation = dict(
    interval=1, metrics=['top_k_accuracy', 'mean_class_accuracy'])

the key-value pairs in the dict are corresponding to EvalHook API.

Migration

Since the key for determine greater an less is related to the downstream task, downstream repos may need to overwrite the self.greater_keys and self.less_keys:

from mmcv.runner import EvalHook as BasicEvalHook

class EvalHook(BasicEvalHook):

    greater_keys = ['a', 'b']
    less_keys = ['c']

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

[dreamerlin]

It can not be merged util #695 is merged since it uses mmcv.engine

[dreamerlin]

A related discuss, open-mmlab/mmaction2#395 (comment). Maybe we need to create rather than symlink

[ZwwWayne]

A related discuss, open-mmlab/mmaction2#395 (comment). Maybe we need to create rather than symlink

How about adding an option for such a case? By default, we create a symlink, but in some cases we can cp a new ckpt. And we should also ensure that only one symlink is created over the whole training time.

[dreamerlin]

A related discuss, open-mmlab/mmaction2#395 (comment). Maybe we need to create rather than symlink

How about adding an option for such a case? By default, we create a symlink, but in some cases we can cp a new ckpt. And we should also ensure that only one symlink is created over the whole training time.

Now I think open-mmlab/mmaction2#395 is a good proposal which cp the ckpt. Since mmcv hook has max_keep_ckpts, which may delete the original ckpt making the symlink for best invalid.

[ZwwWayne]

A related discuss, open-mmlab/mmaction2#395 (comment). Maybe we need to create rather than symlink

How about adding an option for such a case? By default, we create a symlink, but in some cases we can cp a new ckpt. And we should also ensure that only one symlink is created over the whole training time.

Now I think open-mmlab/mmaction2#395 is a good proposal which cp the ckpt. Since mmcv hook has max_keep_ckpts, which may delete the original ckpt making the symlink for best invalid.

I am OK with that.

[ZwwWayne]

Please enrich the PR message for this PR as this is a big refactoring for downstream tasks to allow more discussion and for future reference, including its intention, modification, and consequences.

[ZwwWayne]

1. CI failed.
2. May add the contents/usages in PR messages into the documentation

[dreamerlin]

1. CI failed.
2. May add the contents/usages in PR messages into the documentation

1. CI failed due to mmcv.engine
2. Detailed PR message is updated

[ZwwWayne]

1. May add multi_gpu_test and single_gpu_test in this PR to avoid the dependency on entry point refactoring.
2. Evalhook needs to support synchronizing BN buffer of rank 0 models, see add bn buffer sync in eval_hook mmdetection#4582

[ZwwWayne]

LGTM if the comments can all be resolved.

[dreamerlin]

LGTM if the comments can all be resolved.

Done

[ZwwWayne]

See if @hellock have any comments.

[ZwwWayne]

1. Please resolve conflicts.
2. kindly ping @hellock
3. May need MMAction2 and MMDetection to create a PR based on this PR to verify the correctness.

[LXXXXR]

Validation has been done in mmcls and the performance remaines as it used to be. Thank you.

[wHao-Wu]

Validation has been successfully done in mmdet3d with mmdet's PR: open-mmlab/mmdetection#4806

[congee524]

have tested EvalHook & DistEvalHook on MMAction2.

The related PR "Use mmcv eval hook in mmaction2" will be raised after the release of the next version of mmcv.

[xvjiarui]

Validated in MMSeg. Thx!

[ZwwWayne]

This PR is merged as many downstream repos have verified its correctness. Thanks the efforts of all.