[Feature] Refactor Estimator for computing FLOPs/Params/Latency. #230

gaoyang07 · 2022-08-15T02:28:14Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Refactor Estimator for computing FLOPs/Params/Latency.

Modification

Add ResourceEstimator to estimate model resources.
Refactor mmcv.flops_counter as flops_params_counter.
Add latency_counter.
Add counters for common op counters, e.g. ConvCounter.
Add EstimateResourcesHook.
Add UT for flops_params_counter & ResourceEstimator.
Remove old FlopsEstimator in mmrazor.

BC-breaking (Optional)

Does the modification introduce changes that break the backward compatibility of the downstream repositories?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.

Checklist

Before PR:

Pre-commit or other linting tools are used to fix the potential lint issues.
Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness.
The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects, like MMDet or MMSeg.
CLA has been signed and all committers have signed the CLA in this PR.

1. add EvaluatorLoop in engine.runners; 2. add estimator for structures (both subnet & supernet); 3. add layer_counter for each op.

codecov · 2022-08-15T02:32:28Z

Codecov Report

Merging #230 (77e8095) into dev-1.x (57aec1f) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           dev-1.x    #230      +/-   ##
==========================================
- Coverage     0.48%   0.44%   -0.04%     
==========================================
  Files          144     159      +15     
  Lines         5943    6454     +511     
  Branches       959    1059     +100     
==========================================
  Hits            29      29              
- Misses        5909    6420     +511     
  Partials         5       5

Flag	Coverage Δ
unittests	`0.44% <0.00%> (-0.04%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmrazor/engine/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/engine/hooks/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/engine/hooks/estimate_resources_hook.py	`0.00% <0.00%> (ø)`
mmrazor/engine/runner/autoslim_val_loop.py	`0.00% <0.00%> (ø)`
mmrazor/engine/runner/evolution_search_loop.py	`0.00% <0.00%> (ø)`
mmrazor/engine/runner/slimmable_val_loop.py	`0.00% <0.00%> (ø)`
mmrazor/engine/runner/subnet_sampler_loop.py	`0.00% <0.00%> (ø)`
mmrazor/models/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/task_modules/__init__.py	`0.00% <0.00%> (ø)`
mmrazor/models/task_modules/estimators/__init__.py	`0.00% <0.00%> (ø)`
... and 17 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

sunnyxiaohu · 2022-08-15T03:04:15Z

mmrazor/registry/registry.py

@@ -111,3 +111,6 @@ def build_razor_model_from_cfg(
 VISUALIZERS = Registry('visualizer', parent=MMENGINE_VISUALIZERS)
 # manage visualizer backend
 VISBACKENDS = Registry('vis_backend', parent=MMENGINE_VISBACKENDS)
+
+ESTIMATOR = Registry('estimator')


ESTIMATOR -> ESTIMATORS

sunnyxiaohu · 2022-08-15T03:09:46Z

mmrazor/structures/estimator/latency.py

+        if (i + 1) == max_iter:
+            fps = (i + 1 - num_warmup) / pure_inf_time
+            if PRINT:
+                print(


use logger to print, with debug logger level.

sunnyxiaohu · 2022-08-15T03:11:17Z

mmrazor/structures/estimator/latency.py

+        mean_times_pre_image_ = sum(times_pre_image_list_) / len(
+            times_pre_image_list_)
+        if PRINT:
+            print(


use logger to print, with debug logger level.

sunnyxiaohu · 2022-08-15T03:13:12Z

mmrazor/structures/estimator/base_estimator.py

+
+@ESTIMATOR.register_module()
+class BaseEstimator(metaclass=ABCMeta):
+    """Evaluator for calculating the accuracy and resources consume. Accuracy


update docstring, including necessary Notes.

Add docstring in ResourceEstimator, showing 3 cases when using it.

sunnyxiaohu · 2022-08-15T03:17:58Z

mmrazor/structures/estimator/base_estimator.py

+        self.units = units
+        self.disabled_counters = disabled_counters
+
+    def evaluate(


evaluate -> estimate

1. add ResourceEstimator based on BaseEstimator; 2. add notes & examples for ResourceEstimator & EvaluatorLoop usage; 3. fix a bug of latency test. 4. minor changes according to comments.

sunnyxiaohu · 2022-08-16T03:50:53Z

mmrazor/engine/runner/evaluator_val_loop.py

+        return resource_results
+
+    def export_subnet(self, model):
+        """Export current best subnet."""


update docstring

sunnyxiaohu · 2022-08-16T05:47:19Z

mmrazor/engine/runner/evaluator_val_loop.py

+
+
+@LOOPS.register_module()
+class EvaluatorLoop(ValLoop):


-> ResourceEvaluatorLoop would be better ?

Shall the file name be changed?

yes, and so do the releated UTs.

sunnyxiaohu · 2022-08-16T05:48:48Z

mmrazor/engine/runner/evaluator_val_loop.py

+
+        return resource_results
+
+    def export_subnet(self, model):


is export_subnet sutable for all the NAS alogorithm?

This method is called when it comes to those NAS algorithms that require building a supernet for training. For those algorithms, measuring subnet resources is more meaningful than supernet during validation, therefore this method is required to get the current searched subnet from the supernet.

sunnyxiaohu · 2022-08-16T05:54:27Z

mmrazor/structures/subnet/estimators/flops.py

@@ -93,12 +93,12 @@ class FlopsEstimator:
    def get_model_complexity_info(
            model: Module,
            fix_mutable: Optional[ValidFixMutable] = None,
-            input_shape: Iterable[int] = (3, 224, 224),


delete the directory: subnet/estimators and update the corresponding refs.

sunnyxiaohu · 2022-08-16T05:56:57Z

mmrazor/structures/estimator/resource_estimator.py

+        {'flops': 1.0, 'params': 0.7, 'latency': 0.0}
+
+        >>> # calculate mmrazor.model flops
+        NOTE: check 'EvaluatorLoop' in engine.runner.evaluator_val_loop


add more details for disabled_counters

sunnyxiaohu · 2022-08-16T06:07:34Z

mmrazor/structures/estimator/op_spec_counters/base_counter.py

+from abc import ABCMeta, abstractclassmethod
+
+
+class BaseCounter(object, metaclass=ABCMeta):


Point that XXModuleCounter is responsible for XXModule, which could refers to flops_params_counter::get_counter_type().

humu789

It seems to lack the function of counting flops with the specified scope.
Better not use new registries without parents, it will be not used by other repos of OpenMMLab. You can use directly TASK_UTILS instead of estimator and op_counter
estimator/ is unsuitable to be under structures/. Suggestion location : mmrazor/models/task_modules/.
The file structure of estimator/ could be optimizer. Suggestion:
a. add couters/ in estimator
b. move flops_params_counter.py , latency.py , op_spec_counters/ to counters/
c. rename latency.py to latency_counter.py
d. rename estimator/ to estimators/

humu789 · 2022-08-18T02:37:05Z

mmrazor/registry/registry.py

@@ -111,3 +111,6 @@ def build_razor_model_from_cfg(
 VISUALIZERS = Registry('visualizer', parent=MMENGINE_VISUALIZERS)
 # manage visualizer backend
 VISBACKENDS = Registry('vis_backend', parent=MMENGINE_VISBACKENDS)
+
+ESTIMATORS = Registry('estimator')
+OP_SPEC_COUNTERS = Registry('op_counter')


Better not use new registries without parents, it will be not used by other repos of OpenMMLab. You can use directly TASK_UTILS instead of 'estimator' and 'op_counter'

humu789 · 2022-08-18T03:28:15Z

mmrazor/structures/estimator/resource_estimator.py

+
+        >>> # calculate resources of mmrazor.models
+        NOTE: check 'ResourceEvaluatorLoop' in 
+              engine.runner.resource_evaluator_val_loop for more details.


engine.runner.resource_evaluator_val_loop -> mmrazor.engine.runner.resource_evaluator_val_loop
to avoid ambiguity

humu789 · 2022-08-18T03:49:13Z

mmrazor/engine/runner/resource_evaluator_val_loop.py

+
+
+@LOOPS.register_module()
+class ResourceEvaluatorLoop(ValLoop):


This loop should be for a specific algorithm, you had better name it with the algorithm. It is easy to be misunderstood that ResourceEvaluatorLoop is universal.

ResourceEvaluatorLoop seems to be replaced with Hook, thus we need not maintain source valloop.

Make this part a hook, done.

gaoyang07 · 2022-08-19T07:55:40Z

Now support counting flops with the specified scope. A list of scope names is required from users.

sunnyxiaohu · 2022-08-22T02:26:39Z

mmrazor/engine/runner/evolution_search_loop.py

+        copied_model = copy.deepcopy(self.model)
+        load_fix_subnet(copied_model, fix_mutable)
+
+        estimator = ResourceEstimator()


Use estimator_cfg to build ResourceEstimator， and not use input_shape as fixed kwargs for ::estimate().

sunnyxiaohu · 2022-08-22T02:27:09Z

mmrazor/engine/runner/subnet_sampler_loop.py

+        copied_model = copy.deepcopy(self.model)
+        load_fix_subnet(copied_model, fix_mutable)
+
+        estimator = ResourceEstimator()


Use estimator_cfg to build ResourceEstimator， and not use input_shape as fixed kwargs for ::estimate.

sunnyxiaohu · 2022-08-22T02:43:34Z

mmrazor/models/task_modules/estimators/counters/flops_params_counter.py

+
+def get_model_complexity_info(model,
+                              input_shape,
+                              spec_modules=[],


spec_modules -> custom_keys to support prefix, ref to mmcv::mmcv/runner/optimizer/default_constuctor.py

Now support counting flops with a specified scope, e.g. spec_modules = ['backbone']

sunnyxiaohu · 2022-08-22T02:45:35Z

mmrazor/models/task_modules/estimators/counters/flops_params_counter.py

+    if len(spec_modules):
+        spec_modules_resources = dict()
+        accumulate_sub_module_flops_params(flops_params_model)
+        for name, module in flops_params_model.architecture.named_modules():


Not all the flops_params_model have the architecture attribute.

sunnyxiaohu · 2022-08-22T02:53:07Z

mmrazor/models/task_modules/estimators/counters/flops_params_counter.py

+                precision=precision)) + ' ' + units + 'FLOPs'
+        params_string = str(
+            params_units_convert(
+                accumulated_num_params, units='M',


Unify accumulated_flops_cost and accumulated_num_params with units

A unit pair with FLOPs as 'G' and params as 'M' may be better.

sunnyxiaohu · 2022-08-22T02:55:09Z

mmrazor/models/task_modules/estimators/counters/flops_params_counter.py

+import sys
+from functools import partial
+
+import torch


Unify units for params and flops under the scope of flops_params_counter.py

cancel unit convert in accumulate_sub_module_flops_params, remain the rest as the origin version.

Refactor ModelEstimator:

448c8b9

1. add EvaluatorLoop in engine.runners; 2. add estimator for structures (both subnet & supernet); 3. add layer_counter for each op.

gaoyang07 requested review from sunnyxiaohu, pppppM and humu789 August 15, 2022 02:28

sunnyxiaohu reviewed Aug 15, 2022

View reviewed changes

fix lint

4197ef5

gaoyang07 mentioned this pull request Aug 15, 2022

[Feature] Add Dsnas Algorithm #226

Merged

6 tasks

gaoyang07 added 2 commits August 15, 2022 17:38

update estimator:

43044b5

1. add ResourceEstimator based on BaseEstimator; 2. add notes & examples for ResourceEstimator & EvaluatorLoop usage; 3. fix a bug of latency test. 4. minor changes according to comments.

add UT & fix a bug caused by UT

e8d439e

sunnyxiaohu reviewed Aug 16, 2022

View reviewed changes

gaoyang07 added 3 commits August 16, 2022 20:22

add docstrings & remove old estimator

022dc57

update docstrings for op_spec_counters

39baf6c

rename resource_evaluator_val_loop

bd3679f

gaoyang07 force-pushed the gy/estimator branch from 08e9445 to 91bab7c Compare August 17, 2022 09:16

support adding resource attrs of each submodule in a measured model

15365df

gaoyang07 force-pushed the gy/estimator branch from 91bab7c to 15365df Compare August 17, 2022 09:57

fix lint

bc3ed91

gaoyang07 force-pushed the gy/estimator branch from 0dfe636 to bc3ed91 Compare August 18, 2022 03:44

humu789 reviewed Aug 18, 2022

View reviewed changes

gaoyang07 added 3 commits August 19, 2022 14:50

refactor estimator file structures

be6dd26

support estimating resources for spec modules

6587caf

rm old UT

6cf419e

update new estimator UT cases

3d85a2b

sunnyxiaohu reviewed Aug 22, 2022

View reviewed changes

gaoyang07 added 4 commits August 22, 2022 15:00

fix traversal range of the model

b898a0a

cancel unit convert in accumulate_sub_module_flops_params

335e9c4

use estimator_cfg to build ResourceEstimator

ac5765c

fix a broadcast bug

93af569

gaoyang07 added 5 commits August 22, 2022 17:23

merge dev-1.x into gy/estimator

97ae8c1

delete fixed input_shape

7ea130f

add assertion and string-format-return when measuring spec_modules

f3ef79d

merge 'dev-1.x' into gy/estimator

8564941

add UT for estimating spec_modules

77e8095

sunnyxiaohu merged commit 4b3f8ab into open-mmlab:dev-1.x Aug 23, 2022

humu789 pushed a commit to humu789/mmrazor that referenced this pull request Feb 13, 2023

Fix bug (open-mmlab#230)

64ee8db

		from abc import ABCMeta, abstractclassmethod


		class BaseCounter(object, metaclass=ABCMeta):



		@LOOPS.register_module()
		class ResourceEvaluatorLoop(ValLoop):

[Feature] Refactor Estimator for computing FLOPs/Params/Latency. #230

[Feature] Refactor Estimator for computing FLOPs/Params/Latency. #230

Conversation

gaoyang07 commented Aug 15, 2022 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Aug 15, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sunnyxiaohu Aug 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humu789 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaoyang07 commented Aug 19, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gaoyang07 commented Aug 15, 2022 •

edited

Loading

codecov bot commented Aug 15, 2022 •

edited

Loading

sunnyxiaohu Aug 17, 2022 •

edited

Loading

humu789 left a comment •

edited

Loading