[Fix] fix a bug about validation of ava dataset

[irvingzhang0512]

• Description: When training ava dataset with default configs, no validaton results stored.
• Multi-GPU
  • script: tools/dist_train.sh configs/detection/ava/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb.py 8 --validate
  • details: successfully run eval after every epoch, but best.json is not created in work dir.
• single-gpu:
  • script: python tools/train.py configs/detection/ava/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb.py --validate
  • details: sucessfully run ava eval, but TypeError occured.

/ssd01/zhangyiyang/mmaction2_github/mmaction/core/evaluation/eval_hooks.py:176: UserWarning: The key indicator for evaluation is not included in evaluation result, please specify it in config file
  warnings.warn('The key indicator for evaluation is not '
Traceback (most recent call last):
  File "tools/train.py", line 178, in <module>
    main()
  File "tools/train.py", line 174, in main
    meta=meta)
  File "/ssd01/zhangyiyang/mmaction2_github/mmaction/apis/train.py", line 156, in train_model
    runner.run(data_loaders, cfg.workflow, cfg.total_epochs, **runner_kwargs)
  File "/home/ubuntu/anaconda3/envs/zyy_pytorch1.6/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 125, in run
    epoch_runner(data_loaders[i], **kwargs)
  File "/home/ubuntu/anaconda3/envs/zyy_pytorch1.6/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 54, in train
    self.call_hook('after_train_epoch')
  File "/home/ubuntu/anaconda3/envs/zyy_pytorch1.6/lib/python3.7/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook
    getattr(hook, fn_name)(self)
  File "/ssd01/zhangyiyang/mmaction2_github/mmaction/core/evaluation/eval_hooks.py", line 153, in after_train_epoch
    if (self.save_best and self.compare_func(key_score, self.best_score)):
  File "/ssd01/zhangyiyang/mmaction2_github/mmaction/core/evaluation/eval_hooks.py", line 46, in <lambda>
    rule_map = {'greater': lambda x, y: x > y, 'less': lambda x, y: x < y}
TypeError: '>' not supported between instances of 'NoneType' and 'float'

• Locating BUG: key_indicator for eval hook is not set. Default key_indicator is top1_acc, which is useless for ava dataset.

• Modification:

  • set default key_indicator='[email protected]' for ava dataset configs.
  • rename ava metrics PascalBoxes_Precision/[email protected] to [email protected]
  • modify eval hook, add docs & keys for greater_keys

[innerlee]

Another evidence that our tool scripts need tests

[codecov]

Codecov Report

Merging #527 (17c7664) into master (431004e) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #527   +/-   ##
=======================================
  Coverage   84.67%   84.67%           
=======================================
  Files         121      121           
  Lines        8526     8526           
  Branches     1398     1398           
=======================================
  Hits         7219     7219           
  Misses        953      953           
  Partials      354      354           

Flag	Coverage Δ
unittests	84.65% <100.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...tion/ava_evaluation/object_detection_evaluation.py	65.42% <100.00%> (ø)
mmaction/core/evaluation/eval_hooks.py	83.47% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 431004e...17c7664. Read the comment docs.

[dreamerlin]

LGTM

[kennymckormick]

Two suggestions:

1. What about we use a shorter name like [email protected] instead of the current long name PascalBoxes_Precision/[email protected]? For example, we can change the name in AVADataset.evaluate
2. What about we add the new key_indicator to the greater_keys (here), so that we don't need to wrtie rule='greater' for each config

[irvingzhang0512]

@dreamerlin #395 need to be updated (spatio-temporal action detection related docs and greater_keys)