Replace MobileNetV3's SqueezeExcitation with EfficientNet's one #4487

datumbox · 2021-09-27T11:57:54Z

Partially resolves #4333

All validation stats of models remain the same:

mobilenet_v3_large

torchrun --nproc_per_node=2 train.py --model mobilenet_v3_large --test-only --pretrained

Main Branch:
Test:  Acc@1 74.042 Acc@5 91.340

PR:
Test:  Acc@1 74.042 Acc@5 91.340

mobilenet_v3_small

torchrun --nproc_per_node=2 train.py --model mobilenet_v3_small --test-only --pretrained

Main Branch:
Test:  Acc@1 67.668 Acc@5 87.402

PR:
Test:  Acc@1 67.668 Acc@5 87.402

quantized mobilenet_v3_large

python -u train_quantization.py --device cpu --model mobilenet_v3_large --test-only

Main Branch:
Test:  Acc@1 73.004 Acc@5 90.858

PR:
Test:  Acc@1 73.004 Acc@5 90.858

ssd300_vgg16

torchrun --nproc_per_node=2 train.py --dataset coco --model ssd300_vgg16 --pretrained --test-only

Main Branch:
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.251
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.415
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.262
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.055
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.268
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.239
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.344
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.365
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.088
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.406
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.602

PR:
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.251
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.415
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.262
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.055
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.268
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.435
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.239
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.344
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.365
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.088
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.406
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.602

lraspp_mobilenet_v3_large

torchrun --nproc_per_node=2 train.py --dataset coco --model lraspp_mobilenet_v3_large --pretrained --test-only

Main Branch:
global correct: 91.2
average row correct: ['94.5', '84.3', '69.5', '72.8', '57.7', '42.0', '77.0', '57.0', '90.4', '36.1', '76.0', '60.8', '81.4', '78.9', '81.0', '87.6', '51.3', '83.9', '62.2', '84.2', '56.1']
IoU: ['90.2', '69.2', '57.7', '58.5', '47.8', '35.7', '69.5', '47.1', '79.1', '29.6', '62.6', '34.2', '65.5', '63.4', '70.0', '76.8', '30.1', '61.9', '46.8', '70.6', '49.1']
mean IoU: 57.9

PR:
global correct: 91.2
average row correct: ['94.5', '84.3', '69.5', '72.8', '57.7', '42.0', '77.0', '57.0', '90.4', '36.1', '76.0', '60.8', '81.4', '78.9', '81.0', '87.6', '51.3', '83.9', '62.2', '84.2', '56.1']
IoU: ['90.2', '69.2', '57.7', '58.5', '47.8', '35.7', '69.5', '47.1', '79.1', '29.6', '62.6', '34.2', '65.5', '63.4', '70.0', '76.8', '30.1', '61.9', '46.8', '70.6', '49.1']
mean IoU: 57.9

datumbox

Highlighting some interesting bits of the implementation.

datumbox · 2021-09-27T20:41:41Z

torchvision/models/quantization/mobilenetv3.py

@@ -107,13 +110,13 @@ def _mobilenet_v3_model(
        torch.quantization.prepare_qat(model, inplace=True)

        if pretrained:
-            _load_weights(arch, model, quant_model_urls.get(arch + '_' + backend, None), progress)
+            _load_weights(arch, model, quant_model_urls.get(arch + '_' + backend, None), progress, False)


Earlier versions of the SqueezeExcite class used F.adaptive_avg_pool2d() and F.hardsigmoid() instead of their nn.Module equivalents. Using the latter are advised as because QAT can further optimize them.

Loading the old weights, is still possible but the QAT bits of the above two layers will be missing. Passing strict=false allows us to use the previous weights and achieve the same accuracy.

torchvision/models/mobilenetv3.py

kazhang

LGTM overall. Thanks for working on this!
I only have a question on quantizable module BC.

torchvision/models/quantization/mobilenetv3.py

kazhang

Thanks for consolidating the SE layers!

fmassa

Thanks for the PR!

I've left one comment which I think would be a better way of handling the BC in the quantized model.

I'm approving the PR now, as I would be ok merging the PR as it currently stands.

torchvision/models/quantization/mobilenetv3.py

…one (#4487) Summary: * Reuse EfficientNet SE layer. * Deprecating the mobilenetv3.SqueezeExcitation layer. * Passing the right activation on quantization. * Making strict named param. * Set default params if missing. * Fixing typos. Reviewed By: datumbox Differential Revision: D31270916 fbshipit-source-id: bd10285771f12f61f9b0d0a5487e8ae7aae0a2fc

…rch#4487) * Reuse EfficientNet SE layer. * Deprecating the mobilenetv3.SqueezeExcitation layer. * Passing the right activation on quantization. * Making strict named param. * Set default params if missing. * Fixing typos.

Reuse EfficientNet SE layer.

62dc50a

datumbox added module: models code quality labels Sep 27, 2021

datumbox marked this pull request as draft September 27, 2021 11:58

facebook-github-bot added the cla signed label Sep 27, 2021

datumbox added the module: ops label Sep 27, 2021

Deprecating the mobilenetv3.SqueezeExcitation layer.

72cecb1

datumbox force-pushed the models/replace_se branch 2 times, most recently from 69d462e to e269817 Compare September 27, 2021 20:36

datumbox commented Sep 27, 2021

View reviewed changes

Passing the right activation on quantization.

eedb939

datumbox force-pushed the models/replace_se branch from 8d86c45 to eedb939 Compare September 27, 2021 20:45

Merge branch 'main' into models/replace_se

4969408

datumbox requested a review from kazhang September 27, 2021 20:45

datumbox marked this pull request as ready for review September 27, 2021 20:46

datumbox changed the title ~~[WIP] Replace MobileNetV3's SqueezeExcitation with EfficientNet's one~~ Replace MobileNetV3's SqueezeExcitation with EfficientNet's one Sep 27, 2021

kazhang reviewed Sep 27, 2021

View reviewed changes

torchvision/models/quantization/mobilenetv3.py Outdated Show resolved Hide resolved

torchvision/models/quantization/mobilenetv3.py Show resolved Hide resolved

kazhang approved these changes Sep 28, 2021

View reviewed changes

Making strict named param.

b396443

fmassa approved these changes Sep 29, 2021

View reviewed changes

torchvision/models/quantization/mobilenetv3.py Outdated Show resolved Hide resolved

Set default params if missing.

b491fa2

datumbox force-pushed the models/replace_se branch from b143cf8 to 1851e55 Compare September 29, 2021 13:37

Fixing typos.

24ce2bd

datumbox force-pushed the models/replace_se branch from 1851e55 to 24ce2bd Compare September 29, 2021 13:42

datumbox added 2 commits September 29, 2021 14:44

Merge branch 'main' into models/replace_se

51aaefe

Merge branch 'main' into models/replace_se

dc128a6

datumbox merged commit ff126ae into pytorch:main Sep 29, 2021

datumbox deleted the models/replace_se branch September 29, 2021 14:34

datumbox mentioned this pull request Nov 22, 2021

Fix QuantizableMobileNetV3 weight loading issue #4966

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace MobileNetV3's SqueezeExcitation with EfficientNet's one #4487

Replace MobileNetV3's SqueezeExcitation with EfficientNet's one #4487

datumbox commented Sep 27, 2021 •

edited

Loading

datumbox left a comment

datumbox Sep 27, 2021

kazhang left a comment

kazhang left a comment

fmassa left a comment

Replace MobileNetV3's SqueezeExcitation with EfficientNet's one #4487

Replace MobileNetV3's SqueezeExcitation with EfficientNet's one #4487

Conversation

datumbox commented Sep 27, 2021 • edited Loading

datumbox left a comment

Choose a reason for hiding this comment

datumbox Sep 27, 2021

Choose a reason for hiding this comment

kazhang left a comment

Choose a reason for hiding this comment

kazhang left a comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

datumbox commented Sep 27, 2021 •

edited

Loading