Unify Tensor and PIL transforms

[fmassa]

Now that most of the tensor transforms have been implemented #1375, it is time to make unify the implementations for PIL and Tensor so that torchvision.transforms and torchvision.transforms.functional works seamlessly between both datatypes.

As such, we would like that the following work with the same interface:

# functional interface
torchvision.transforms.functional.hflip(pil_image)
torchvision.transforms.functional.hflip(tensor_image)

# class interface
transform = torchvision.transforms.RandomHorizontalFlip(0.3)
transform(pil_image)
transform(tensor_image)

# torchscript support for tensor_image
script_transform = torch.jit.script(transform)
script_transform(tensor_image)

Example PRs adding support for hflip and vflip: #2282 #2283

We would like to add support for torchscript to the torchvision.transforms interface as well, which for simplicity might require a few changes. Most notably we would need the transforms to inherit from nn.Module.

Here is a list of transforms that can be readily converted:

• adjust_brightness
• adjust_contrast
• adjust_saturation
• crop
• center_crop
• five_crop
• ten_crop
• pad (Added symmetric padding mode for Tensors #2373 Unified Pad and F.pad opertion for PIL and Tensor inputs #2345)
• resize (Unified input for resize op  #2394)
• resized_crop (Unified input for resized crop op #2396)
• perspective ([BC-breaking] Unified input for F.perspective #2558, [BC-breaking] Unified input for RandomPerspective #2561)
• adjust_hue (adjust_hue now supports inputs of type Tensor #2566)
• adjust_gamma (Add torchscriptable adjust_gamma transform #2459)
• rotate (Unified inputs for F.rotate #2495 Unified inputs for T.RandomRotation #2496 Fixes F.affine and F.rotate to support rectangular tensor images #2553)
• affine (Unified input for F.affine #2444, Fixes incoherence in affine transformation with center is defined as … #2468, Unified inputs for T.RandomAffine transformation (2292) #2478, Fixes F.affine and F.rotate to support rectangular tensor images #2553)
• to_grayscale

[clemkoa]

Hi! I would be glad to work on a PR for this issue. I can start with the color jitter if that's okay with you

[vfdev-5]

@fmassa I can work on some of these listed transformation if it is OK.

[dimimal]

In the documentation of pytorch it is mentioned that functional module works with both tensors and pil objects. This is about torch version 1.5.1 and torchvision 0.6.1. However, when I try to use them with torch.tensor it throws an error that the object is not PIL. I checked the source code and the snippet shown in the docs is missing. I use the functional.hflip() function of the docs. I thought that the unification has been completed, hasn't it?

[fmassa]

@dimimal the unification is not yet complete

Can you point out where in the documentation you found the reference that all functions work with both tensors and PIL Images?

[fmassa]

@vfdev-5 sure go for it!

[dimimal]

@fmassa check this https://pytorch.org/docs/stable/_modules/torchvision/transforms/functional.html#hflip

[fmassa]

@dimimal hflip supports both tensors and PIL Images since #2282 , but only on master and only a handful of functions support it for now

[dimimal]

@fmassa I wasn't using the master and the docs confused me with the noted version. Thank you. Is there anything I can do to help in contribution?

[fmassa]

@dimimal sure, contributions are more than welcome! There are two steps here: finishing up #1375 , and then implementing the glue code, similarly to the example PRs #2282 #2283, or more recently #2345 I think implementing adjust_gamma would be a good issue to start with.

Note that one of the biggest difficulties will be making the implementation work with torchscript, as not everything that we use is natively supported.

[vfdev-5]

Complete list for ops to support:

• adjust_brightness
• adjust_contrast
• adjust_saturation
• crop
• center_crop
• five_crop
• ten_crop
• pad (Added symmetric padding mode for Tensors #2373 Unified Pad and F.pad opertion for PIL and Tensor inputs #2345)
• resize (Unified input for resize op  #2394)
• resized_crop (Unified input for resized crop op #2396)
• perspective ([BC-breaking] Unified input for F.perspective #2558, [BC-breaking] Unified input for RandomPerspective #2561 self-assigned)
• adjust_hue (@CristianManta )
• adjust_gamma (Add torchscriptable adjust_gamma transform #2459)
• rotate (Unified inputs for F.rotate #2495 Unified inputs for T.RandomRotation #2496 Fixes F.affine and F.rotate to support rectangular tensor images #2553 self-assigned)
• affine (Unified input for F.affine #2444, Fixes incoherence in affine transformation with center is defined as … #2468, Unified inputs for T.RandomAffine transformation (2292) #2478, Fixes F.affine and F.rotate to support rectangular tensor images #2553, self-assigned)
• to_grayscale (self-assigned + already existing rgb_to_grayscale)
• RandomErasing as torch.nn.module ([BC-breaking] RandomErasing is now scriptable #2386)

[vfdev-5]

@fmassa if ops should also work on tensors on non-cpu devices, a supplementary check of certain geometrical ops will be needed. Should we setup some CI tests for that ?

[fmassa]

Should we setup some CI tests for that ?

Yes, please!

[vfdev-5]

@nairbv would you be interested in working on unifying adjust_hue ?

[fmassa]

@vfdev-5 go for it!

[vfdev-5]

According to a discussion with @fmassa , we also have to make sure that all transformations work with a batch of images.

EDIT: requested as a another issue : #2583

[vfdev-5]

The issue can be closed as all mentioned transformations now support tensor and pil image as input.

[voldemortX]

Hi! Very exciting codes! Wondering whether future commits will support fill color other than 0 for tensor affine transformations? It is kind of important for tasks like semantic segmentation when transforming labels. @vfdev-5

[vfdev-5]

@voldemortX thanks for the feedback !

Wondering whether future commits will support fill color other than 0 for tensor affine transformations? It is kind of important for tasks like semantic segmentation when transforming labels.

Could you please open an issue for that such we could track this feature request and see what could be possible to do.
Currently, the blocker is with pytorch grid_sample which has

padding_mode="zeros": use 0 for out-of-bound grid locations

[voldemortX]

@voldemortX thanks for the feedback !

Wondering whether future commits will support fill color other than 0 for tensor affine transformations? It is kind of important for tasks like semantic segmentation when transforming labels.

Could you please open an issue for that such we could track this feature request and see what could be possible to do.
Currently, the blocker is with pytorch grid_sample which has

padding_mode="zeros": use 0 for out-of-bound grid locations

New issue opened #2887