Add Quantizable MobilenetV3 architecture for Classification

[datumbox]

The pre-trained model was trained:

python -m torch.distributed.launch --nproc_per_node=8 --use_env train_quantization.py\
--model='mobilenet_v3_large' --wd 0.00001 --lr 0.001

Submitted batch job 35496554

Validated with:

python train_quantization.py --device='cpu' --model='mobilenet_v3_large' --test-only

Accuracy metrics (Epoch 89):
Acc@1 73.004 Acc@5 90.858

Speed Benchmark: 0.0162 sec per image on CPU

[codecov]

Codecov Report

Merging #3323 (bc27744) into master (859a535) will increase coverage by 0.13%.
The diff coverage is 92.85%.

@@            Coverage Diff             @@
##           master    #3323      +/-   ##
==========================================
+ Coverage   73.90%   74.04%   +0.13%     
==========================================
  Files         104      105       +1     
  Lines        9618     9692      +74     
  Branches     1544     1554      +10     
==========================================
+ Hits         7108     7176      +68     
- Misses       2028     2033       +5     
- Partials      482      483       +1     

Impacted Files	Coverage Δ
torchvision/models/quantization/mobilenetv3.py	92.42% <92.42%> (ø)
torchvision/models/mobilenetv3.py	92.42% <93.33%> (-0.38%)	⬇️
torchvision/models/quantization/mobilenet.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 859a535...274c6a1. Read the comment docs.

[fmassa]

Looks great, thanks a lot!

I have a few questions to @raghuramank100 which are not blocking to get this PR merged, but I would love to get this thoughts on a couple of points.

[fmassa]

@raghuramank100 the approach we had to follow here for loading the pre-trained weights for a quantized model is different from what we did for the other models. Would you happen to know why we can't use the previous approach

[raghuramank100]

Can you elaborate on why the other approach didnt work? For QAT, we should load the fp32 model (prior to prepare) and then start training with that.

[datumbox]

For QAT, we should load the fp32 model (prior to prepare) and then start training with that.

We did not face any issues during the training process and what you describe is the approach we used for training the model.

Can you elaborate on why the other approach didnt work?

After the training was completed, we tried to load the weights of the quantized model (key "model_eval" of the checkpoint) to do inference. Unfortunately doing so leads to extremely low accuracy (less than 1%). We are certain that the weights are loaded on the model.

As a workaround, we opted for loading the weights of the QAT model (key "model"` of the checkpoint) and then convert it. This works fine and gives the same inference accuracy as observed during training.

We are trying to understand what could be the reason for this behaviour. I can provide a demo script if that helps.

[fmassa]

@raghuramank100 this is ~ 1 acc@1 point drop compared to the fp32 reference. Would you have any tips on how to make this gap smaller?

[datumbox]

@fmassa The non-quantized version of MobileNet V3 Large uses averaging of checkpoints which I don't do here. That's possibly one of the reasons we get lower accuracy.

[raghuramank100]

If you start with the averaged checkpoint to start quantization aware training, you should get better accuracy as the starting point is better.

[raghuramank100]

Also, one additional hyper-parameter that helps is to turn on QAT in steps: We first turn observers on (i.e collect statistics) and then turn fake-quantization on, and after sometime we turn batch norm off. Currently, in train_quantization, steps 1 and 2 are combined. We have seen that separating them helps with QAT accuracy in some models. You could try something like:

# Initially only turn on observers, disable fake quant
model.apply(torch.quantization.enable_observer)
model.apply(torch.quantization.disable_fake_quant)
....

 if epoch >= args.num_fake_quant_start_epochs:
   model.apply(torch.quantization.enable_fake_quant)
 if epoch >= args.num_observer_update_epochs:
                print('Disabling observer for subseq epochs, epoch = ', epoch)
                model.apply(torch.quantization.disable_observer)
            if epoch >= args.num_batch_norm_update_epochs:
                print('Freezing BN for subseq epochs, epoch = ', epoch)
                model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)

[datumbox]

If you start with the averaged checkpoint to start quantization aware training, you should get better accuracy as the starting point is better.

We indeed start from an averaged checkpoint but that's not what I mean here. I'm referring to the post-training averaging step which is missing.

We first turn observers on (i.e collect statistics) and then turn fake-quantization on.

That's worth integrating on the new quant training script.

I believe key reason why the accuracy is lagging is because the quant training script does not currently support all the enhancements made on the classification training script. These enhancements (Multiple restarts, Optimizer tuning, Data augmentation, model averaging at the end etc) helped me push the accuracy by 2 points.

[fmassa]

BTW, test failures seem to be related

[datumbox]

@fmassa Thanks for flagging. I fixed the issues, did 1 epoch of training and 1 validation to ensure everything still works fine. The rest of the failing tests are not related to this PR. So we should be OK.

[raghuramank100]

One suggestion for improvement is to change the qconfig to the following: This configuration uses per-channel quantization for weights with qnnpack, which is now supported.
qconfig = QConfig(activation=torch.quantization.FakeQuantize.with_args(observer=torch.quantization.MovingAverageMinMaxObserver,
quant_min=0,
quant_max=255,
reduce_range=False),
weight= default_per_channel_weight_fake_quant)