TorchVision support for pre-trained, quantised, ResNet50 using QNNpack backend

[nSircombe]

🚀 Feature

Addition of pre-trained, quantised ResNet50 model to torchvision.models with support for QNNpack backend.

Motivation

Quantised models have been shown to deliver greater performance than their fp32 equivalents without sacrificing accuracy.
The quantised ResNet50 model currently available in TorchVision supports the FBGEMM backend, but does not provide support for QNNpack. On AArch64, only the QNNpack backend is currently supported, so expanding the functionality of the ResNet50 model would allow its use one AArch64 platforms.
Expanding the capabilities of the quantised models available to TorchVision users would provide functionality equivalent to what is currently available in other frameworks.

Pitch

Quantised, pre-trained, ResNet50 models are available for use on with TensorFlow (https://github.com/IntelAI/models/blob/master/benchmarks/image_recognition/tensorflow/resnet50v1_5/README.md#int8-inference-instructions) and typically deliver improved performance over their fp32 equivalents.

Are there any plans to expand the supported backends for the quantised ResNet50 model to TorchVision to cover QNNpack?

Alternatives

Where the workload or benchmark requires ResNet50, the alternative is to make your own quantised model using the qnnpack backend.

[oke-aditya]

Yes ! Torchvision does provide Quantized models, such as quantized resnet50. All the models are int8 quantized.

Documentation for these is available over master. Visit here

I think they are not reflected in current website documentation, but with next release, it will be updated.

These models were introduced a year ago with 0.5.0 release have a read here

[nSircombe]

Hi @oke-aditya,

Thanks for your quick response. I'm sorry I should have been clearer in my original query, I'm curious about quantised ResNet50 models and accompanying support for a quantised backend. At present if I use mobilenet_v2 I'm able to get a quantised model which uses the QNNpack backend (on AArch64)...

>>> import torchvision.models as models
>>> model = models.quantization.mobilenet_v2(pretrained=True, quantize=True)

However, this doesn't appear to be the case for ResNet50...

>>> import torchvision.models as models
>>> resnet50 = models.quantization.resnet50()
>>> resnet50 = models.quantization.resnet50(pretrained=True, quantize=True)
RuntimeError: Quantized backend not supported 

So it appears that support for a quantised backend is currently missing for ResNet50, is that the case? If so are there any plans to expand support?

[oke-aditya]

There are two Quantization backend in PyTorch fbgemm and QNNpack.
As you pointed out mobilenet_v2 is available with QNNpackbackend.

The resnet50 model it is available with fbgemm backend. Have a look here. I hope it will work with fbgemm backend and probably you shouldn't get errors with fbgemm.

I think you are very right that models are not available for both quantization backends. Also that some are available for QNNpack while some are for fbgemm.

Yes I think support for both backends for all models isn't there yet. I'm not sure of any plans. Maybe it is plan of Roadmap #3221

[nSircombe]

Thanks @oke-aditya, I didn't find any details of QNNpack support for ResNet50 in the plans, but have asked in-thread.

I've updated my 'Feature Request' above in light of our discussion, so it's more specific to my use-case (QNNpack backend, on AArch64).

[fmassa]

Thanks for the request and sorry for the delay in replying.

I might not remember correctly anymore, but IIRC we provided fbgemm backends for ResNet because fbgemm was better suited for server inference, and ResNet50 was normally too big for running on embedded devices (on which qnnpack was better).

@nSircombe are you planning on running ResNet50 on embedded devices, or is it that fbgemm doesn't work in your system?