Adding RandAugment implementation #4348

datumbox · 2021-09-01T17:49:10Z

Partially implements #3817

Inspired from work started by @SamuelGabriel at #4221

cc @vfdev-5 @datumbox

datumbox

Below I highlight a few bits of the implementation that I think is noteworthy.

@SamuelGabriel Given your work on the area I would love it if you can provide your input once you are back.

datumbox · 2021-09-01T19:00:57Z

torchvision/transforms/autoaugment.py

+            # op_name: (magnitudes, signed)
+            "ShearX": (torch.linspace(0.0, 0.3, num_bins), True),
+            "ShearY": (torch.linspace(0.0, 0.3, num_bins), True),
+            "TranslateX": (torch.linspace(0.0, 150.0 / 331.0 * image_size[0], num_bins), True),


@SamuelGabriel I noticed in your implementation you restrict the maximum value of Translate to 14.4. I couldn't find a reference of that on the RandAugment paper. I was hoping if you could provide some background info?

I did not use this setting for my RA experiments actually. See for example here: https://github.com/automl/trivialaugment/blob/master/confs/wresnet28x10_svhncore_b128_maxlr.1_ra_fixed.yaml

I used fixed_standard which is the search space described in the RA paper, but slightly different from that in the implementation of the significant parts of this (https://github.com/tensorflow/models/tree/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment) AutoAugment implementation (which RA follows with its augmentation space), therefore I call it fixed. The setting you speak about is for their ImageNet experiment, as my re-implementations of AA/RA where on 32x32 images (CIFAR/SVHN), I followed the implementation above. Here they set the translation to 10: https://github.com/tensorflow/models/blob/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment/augmentation_transforms.py#L319 They do not use the same augmentation space across datasets...

I am actually not sure what is the best strategy to follow here. Any idea? The same problem arises for AutoAugment actually. Should we ask the authors or focus only on 32x32 images or only on ImageNet? For TA it is simpler we use the same setting across datasets.

Not 100% sure either. Here I decided to use this approach because of their comment on Table 6 on the AA paper. This also means that if you are a 32x32 image as in the case of CIFAR, your Translate max value would be 14.5, which is similar but on equal to yours and hence it sparked my interest on how you derived it. Not sure if this subpixel diff matters here.

datumbox · 2021-09-01T19:02:53Z

torchvision/transforms/autoaugment.py

@@ -178,9 +178,9 @@ def _get_transforms(
        else:
            raise ValueError("The provided policy {} is not recognized.".format(policy))

-    def _get_magnitudes(self, num_bins: int, image_size: List[int]) -> Dict[str, Tuple[Tensor, bool]]:
+    def _augmentation_space(self, num_bins: int, image_size: List[int]) -> Dict[str, Tuple[Tensor, bool]]:


Even thought this is a private method, I decided to use the terminology of the TrivialAugment paper as I think it describes better what we get back (combination of permitted ops and magnitudes) for the given augmentation.

datumbox · 2021-09-01T19:07:26Z

torchvision/transforms/autoaugment.py

+            image. If given a number, the value is used for all bands respectively.
+        """
+
+    def __init__(self, num_ops: int = 2, magnitude: int = 9, num_magnitude_bins: int = 30,


N=2, M=9 are the best ImageNet values for ResNet50 (see A.2.3 on paper).

num_magnitude_bins=30 because the majority of the experiments on the paper used this value. Weirdly section A.2.3 mentions trying the level 31 for EfficientNet B7.

The num_magnitude_bins should be 31, like for TA, as 0 is also a bin and in the paper the maximal value is 30. That they tried level 31 is definitely weird and, I guess, a typo.

fmassa

Looks great, thanks!

I haven't checked the bin values for the augmentation space, but the rest is good with me.

I've made a couple of minor comments, none of which are merge blocking

fmassa · 2021-09-02T12:03:40Z

torchvision/transforms/autoaugment.py

+        if isinstance(img, Tensor):
+            if isinstance(fill, (int, float)):
+                fill = [float(fill)] * F.get_image_num_channels(img)
+            elif fill is not None:
+                fill = [float(f) for f in fill]


nit: might be worth putting this in a helper function, or push it directly to _apply_op maybe?

Good call, I'll add this in a helper method for now. This can move to a base class once we nail the API.

The JIT is giving me headaches if I move it on ops. I'll add a TODO to remove duplicate code once we have a base class.

fmassa · 2021-09-02T12:06:26Z

torchvision/transforms/autoaugment.py

@@ -239,3 +239,87 @@ def forward(self, img: Tensor) -> Tensor:

    def __repr__(self) -> str:
        return self.__class__.__name__ + '(policy={}, fill={})'.format(self.policy, self.fill)
+
+
+class RandAugment(torch.nn.Module):


I wonder if it makes sense to inherit from AutoAugment and override only the _augmentation_space method?

Yes indeed. It's a direct copy-paste. The only reason I didn't make it static or inherit from AutoAugment is because I think we haven't nailed the API of the base class yet. I was thinking of keeping only the public parts visible and make changes once we add a couple of methods.

SamuelGabriel

To me it looks good, besides the mentioned problems of the augmentation space.

SamuelGabriel · 2021-09-06T07:40:46Z

torchvision/transforms/autoaugment.py

+            image. If given a number, the value is used for all bands respectively.
+        """
+
+    def __init__(self, num_ops: int = 2, magnitude: int = 9, num_magnitude_bins: int = 30,


The num_magnitude_bins should be 31, like for TA, as 0 is also a bin and in the paper the maximal value is 30. That they tried level 31 is definitely weird and, I guess, a typo.

SamuelGabriel · 2021-09-06T07:53:52Z

torchvision/transforms/autoaugment.py

+            # op_name: (magnitudes, signed)
+            "ShearX": (torch.linspace(0.0, 0.3, num_bins), True),
+            "ShearY": (torch.linspace(0.0, 0.3, num_bins), True),
+            "TranslateX": (torch.linspace(0.0, 150.0 / 331.0 * image_size[0], num_bins), True),


I did not use this setting for my RA experiments actually. See for example here: https://github.com/automl/trivialaugment/blob/master/confs/wresnet28x10_svhncore_b128_maxlr.1_ra_fixed.yaml

I used fixed_standard which is the search space described in the RA paper, but slightly different from that in the implementation of the significant parts of this (https://github.com/tensorflow/models/tree/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment) AutoAugment implementation (which RA follows with its augmentation space), therefore I call it fixed. The setting you speak about is for their ImageNet experiment, as my re-implementations of AA/RA where on 32x32 images (CIFAR/SVHN), I followed the implementation above. Here they set the translation to 10: https://github.com/tensorflow/models/blob/fd34f711f319d8c6fe85110d9df6e1784cc5a6ca/research/autoaugment/augmentation_transforms.py#L319 They do not use the same augmentation space across datasets...

SamuelGabriel · 2021-09-06T07:59:24Z

torchvision/transforms/autoaugment.py

+            "Solarize": (torch.linspace(256.0, 0.0, num_bins), False),
+            "AutoContrast": (torch.tensor(0.0), False),
+            "Equalize": (torch.tensor(0.0), False),
+            "Invert": (torch.tensor(0.0), False),


Invert should not be here, but Identity should be. I believe you replicated the mistake I made in the TA implementation for Vision. Sorry for that. I'll fix it there, too. TA and RA use the same augmentation operations.

Ah, thanks! Indeed this was copied from you. Your implementation heavily inspired how the entire code was refactored here, so thanks a lot for the contribution.

SamuelGabriel · 2021-09-06T08:02:14Z

torchvision/transforms/autoaugment.py

+            # op_name: (magnitudes, signed)
+            "ShearX": (torch.linspace(0.0, 0.3, num_bins), True),
+            "ShearY": (torch.linspace(0.0, 0.3, num_bins), True),
+            "TranslateX": (torch.linspace(0.0, 150.0 / 331.0 * image_size[0], num_bins), True),


I am actually not sure what is the best strategy to follow here. Any idea? The same problem arises for AutoAugment actually. Should we ask the authors or focus only on 32x32 images or only on ImageNet? For TA it is simpler we use the same setting across datasets.

datumbox

@SamuelGabriel Thanks a lot for your review. I'll fix the issues you highlighted on a separate PR as this is already merged.

datumbox · 2021-09-06T08:55:11Z

torchvision/transforms/autoaugment.py

+            # op_name: (magnitudes, signed)
+            "ShearX": (torch.linspace(0.0, 0.3, num_bins), True),
+            "ShearY": (torch.linspace(0.0, 0.3, num_bins), True),
+            "TranslateX": (torch.linspace(0.0, 150.0 / 331.0 * image_size[0], num_bins), True),


Not 100% sure either. Here I decided to use this approach because of their comment on Table 6 on the AA paper. This also means that if you are a 32x32 image as in the case of CIFAR, your Translate max value would be 14.5, which is similar but on equal to yours and hence it sparked my interest on how you derived it. Not sure if this subpixel diff matters here.

datumbox · 2021-09-06T08:59:35Z

torchvision/transforms/autoaugment.py

+            "Solarize": (torch.linspace(256.0, 0.0, num_bins), False),
+            "AutoContrast": (torch.tensor(0.0), False),
+            "Equalize": (torch.tensor(0.0), False),
+            "Invert": (torch.tensor(0.0), False),


Ah, thanks! Indeed this was copied from you. Your implementation heavily inspired how the entire code was refactored here, so thanks a lot for the contribution.

Summary: Pull Request resolved: facebookresearch/vissl#421 * Adding randaugment implementation * Refactoring. * Adding num_magnitude_bins. * Adding FIXME. Reviewed By: fmassa Differential Revision: D30793331 fbshipit-source-id: 7a99c6d2e64931e10672ceea9e81309c62a799af

Adding randaugment implementation

ecdd9ee

datumbox added the module: transforms label Sep 1, 2021

facebook-github-bot added the cla signed label Sep 1, 2021

datumbox marked this pull request as draft September 1, 2021 17:49

datumbox mentioned this pull request Sep 1, 2021

[RFC] New Augmentation techniques in Torchvison #3817

Open

17 tasks

datumbox and others added 3 commits September 1, 2021 19:50

Refactoring.

89e73f1

Adding num_magnitude_bins.

bd619da

Merge branch 'main' into transforms/randaugment

19ddfbf

datumbox commented Sep 1, 2021

View reviewed changes

datumbox marked this pull request as ready for review September 1, 2021 19:12

datumbox requested a review from fmassa September 1, 2021 19:13

datumbox added the new feature label Sep 2, 2021

datumbox changed the title ~~[WIP] Adding RandAugment implementation~~ Adding RandAugment implementation Sep 2, 2021

fmassa approved these changes Sep 2, 2021

View reviewed changes

datumbox and others added 2 commits September 2, 2021 13:29

Adding FIXME.

5b8d889

Merge branch 'main' into transforms/randaugment

d173ac4

datumbox merged commit 5a81554 into pytorch:main Sep 2, 2021

datumbox deleted the transforms/randaugment branch September 2, 2021 12:31

datumbox mentioned this pull request Sep 2, 2021

Adding docs for RandAugment #4349

Merged

datumbox mentioned this pull request Sep 4, 2021

[RFC] TorchVision with Batteries included - Phase 1 #3911

Closed

16 tasks

SamuelGabriel reviewed Sep 6, 2021

View reviewed changes

datumbox commented Sep 6, 2021

View reviewed changes

This was referenced Sep 6, 2021

Integration of TrivialAugment with the current AutoAugment Code #4221

Merged

Fix RandAugment and TrivialAugment bugs #4370

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding RandAugment implementation #4348

Adding RandAugment implementation #4348

datumbox commented Sep 1, 2021 •

edited by pytorch-probot bot

Loading

datumbox left a comment

datumbox Sep 1, 2021 •

edited

Loading

SamuelGabriel Sep 6, 2021

SamuelGabriel Sep 6, 2021

datumbox Sep 6, 2021

datumbox Sep 1, 2021 •

edited

Loading

datumbox Sep 1, 2021

SamuelGabriel Sep 6, 2021

fmassa left a comment

fmassa Sep 2, 2021

datumbox Sep 2, 2021

datumbox Sep 2, 2021

fmassa Sep 2, 2021

datumbox Sep 2, 2021

SamuelGabriel left a comment

SamuelGabriel Sep 6, 2021

SamuelGabriel Sep 6, 2021

SamuelGabriel Sep 6, 2021

datumbox Sep 6, 2021

SamuelGabriel Sep 6, 2021

datumbox left a comment

datumbox Sep 6, 2021

datumbox Sep 6, 2021

Adding RandAugment implementation #4348

Adding RandAugment implementation #4348

Conversation

datumbox commented Sep 1, 2021 • edited by pytorch-probot bot Loading

datumbox left a comment

Choose a reason for hiding this comment

datumbox Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox Sep 1, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fmassa left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SamuelGabriel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

datumbox commented Sep 1, 2021 •

edited by pytorch-probot bot

Loading

datumbox Sep 1, 2021 •

edited

Loading

datumbox Sep 1, 2021 •

edited

Loading