TorchVision Roadmap - 2021 H1

[datumbox]

This issue is be used to keep track of the roadmap of TorchVision and get feedback from our users.

For the 2021-H2 roadmap, please see #4187

• Streamlined data pipelines:
  • Improved io.image with new supported types and conversions (Extend the supported types of decodePNG #2984, Support specifying output channels in io.image.read_image #2988, Add support of mode and remove channels #3024, Check num of channels on adjust_* transformations #3069)
  • Add Auto-augment input transformation (Implement all AutoAugment transforms + Policies #3123)
  • Add preset preprocessing transforms on pretrained models (Adding Preset Transforms in reference scripts #3317)
  • Add GPU/CUDA ops for image decoding (jpeg) (Support for decoding jpegs on GPU with nvjpeg #3792)
  • Add and optimize native uint8 support for interpolate in PyTorch
• Mobile support:
  • Refactor C++ codebase & make it mobile-friendly (Static Analysis corrections on DeformConv #2885, Static Analysis corrections on image/  #2891, Static Analysis corrections on *ROIAlign #2892, Static Analysis corrections on c++ model classes #2893, Define all C++ model constructors explicit #2944, Replacing C-style typedefs with aliases. #2945, Removing python headers and Win32 Python2 inits in C++ #3011, Encapsulate and Standardise C++ Ops #3097, Per file C++ Operator registration #3135, Per file C++ Operator registration #3135, Restructuring C++ project #3146, [BC-breaking] Remove _new_empty_tensor op #3156, Move autograd implementations on separate files. #3154, Removing VISION_API from backward() methods and adding an ops.h #3163, Add Torch Selective macros in all C++ Ops for better support on mobile #3218, Restructure the video/video_reader C++ codebase #3311, Add namespace on image C++ codebase #3312)
  • Add mobile-ready models for detection (Add MobileNetV3 architecture for Detection #3253, Add High-res FasterRCNN MobileNetV3 and tune Low-res for speed #3265, Add SSD architecture with VGG16 backbone #3403, Add SSDlite architecture with MobileNetV3 backbones #3757, Add SSD512 with ResNet50 backbone #3760)
  • Add mobile-ready models for segmentation (Add MobileNetV3 architecture for Segmentation #3276)
  • Support D2Go OSS
  • Add quint8 implementations for nms and RoIAlign (Add quantized version of nms #3601, Add Quantized version of RoIAlign #3624)
  • Rotated Boxes ([RFC] Rotated Bounding Boxes #2761) (moved to 2022H2 TorchVision Roadmap - 2021 H2 #4187)
• Add newer models for classification:
  • MobileNetV3 (Add MobileNetV3 architecture for Classification #3252, Adding MobileNetV3 Small pre-trained model #3354)
  • MobileNetV3 Quantized (Add Quantizable MobilenetV3 architecture for Classification #3323)

[masahi]

Can we revisit various workarounds for tracing Faster RCNN / MaskRCNN ? There are many such special code-path for tracing mode, to support ONNX export. These code were added one year ago when PyTorch ONNX had many limitation. Especially the one below does repeated concat in a loop so it is not efficient.

vision/torchvision/models/detection/roi_heads.py

Lines 454 to 461 in 3d60f49

	@torch.jit._script_if_tracing
	def _onnx_paste_masks_in_image_loop(masks, boxes, im_h, im_w):
	res_append = torch.zeros(0, im_h, im_w)
	for i in range(masks.size(0)):
	mask_res = _onnx_paste_mask_in_image(masks[i][0], boxes[i], im_h, im_w)
	mask_res = mask_res.unsqueeze(0)
	res_append = torch.cat((res_append, mask_res))
	return res_append

Now that PyTorch ONNX export has support for lists and also inplace assignment, I think many of those workarounds can be removed. Even if removing is not possible, we should be able to make them more efficient, thanks to the improved ONNX export support in recent PyTorch. For example, The snippet above can be replaced with the one below, which uses a batched concat. I confirmed that ONNX export of maskrcnn works with this change applied. This is also what TVM prefers.

@torch.jit._script_if_tracing
def _onnx_paste_masks_in_image_loop(masks, boxes, im_h, im_w):
    res = []
    for i in range(masks.size(0)):
        mask_res = _onnx_paste_mask_in_image(masks[i][0], boxes[i], im_h, im_w)
        res += [mask_res]
    return torch.stack(res, dim=0)[:, None]

cc @fmassa @lara-hdr

[fmassa]

@masahi I would be happy to accept a PR to improve the ONNX export path, and removing as many workarounds as possible would be awesome. I've opened an issue in #3225 to centralize this discussion there.

[oke-aditya]

Just linking a few trackers here

• New Models
• Loss Functions
• New Datasets
• New Transforms

[vfdev-5]

How about adding RandAugment, in addition to already implemented AutoAugment ?

[nSircombe]

Hi,

I was just wondering if there are any plans to add support for the QNNpack backend to the quantised ResNet50 model? (this would enable quantised ResNet50 for AArch64, which isn't supported by the FBGEMM backend).

[zhiqwang]

• Mobile support:
  • Support D2Go OSS

Is the newly released d2go mentioned here?

[oke-aditya]

Thoughts about FastAutoAugment ?

[jamt9000]

• Add GPU/CUDA ops for image decoding (jpeg)

Implemented in #3792

[oke-aditya]

Rotated Boxes RFC #2761.

[fmassa]

@nSircombe

I was just wondering if there are any plans to add support for the QNNpack backend to the quantised ResNet50 model? (this would enable quantised ResNet50 for AArch64, which isn't supported by the FBGEMM backend).

can you please open a new issue so that we can track this?

@zhiqwang

Is the newly released d2go mentioned here?

yes, exactly

@oke-aditya

Thoughts about FastAutoAugment ?

Can you open a new issue so that we can track it down and discuss it there?

[nSircombe]

Hi @fmassa,

I currently have #3362 open (I should have added this in my original comment, sorry), will that suffice?

[Atze00]

Is there any interest in adding MoViNets or some other state of the art models in video understanding?
Are models valuable also if they are not pretrained/someone has the possibility to train them before pushing to Pytorch?
Otherwise these models may use imported weights from tf, with the overhead of same padding, but I'm not convinced about this option.

[NicolasHug]

I updated the post with our roadmap for H2. Actually, I created a new issue #4187 to avoid mixing discussions

Comments and suggestions are welcome :) !

[NicolasHug]

@Atze00 there's no current plan to support MoViNets but regarding video understanding, we're planning to support Optical Flow with a RAFT implementation

Are models valuable also if they are not pretrained/someone has the possibility to train them before pushing to Pytorch?

Could you clarify what you mean? I'm not sure this answers your question but in general, when we implement a model in torchvision we provide the architecture, the pre-trained weights, and also some training receipe

[Atze00]

@NicolasHug My question was based on the fact that I've been working on implementing movinets in pytorch, using the weights released by the authors in TF. I've been wondering if the code alone would be enough to discuss about a possible addition of this architecture into the roadmap. Unfortunately I don't have the resources or the experience necessary to reproduce state of the art results in video understanding.

[datumbox]

@fmassa @NicolasHug Shall we close this issue in favor of #4187?

Since only the rotated boxes RFC is still open we could consider moving it on H2 or omit it if nobody wants to pick it up.

[datumbox]

Moved the pending to H2 at #4187, closing this issue.