Add MobileNetV3 Architecture in TorchVision

[datumbox]

Partially fixes #1676

Depends and cherrypicks commits from #3177

---

The current temporary pre-trained model was trained:

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
--model mobilenet_v3_large --epochs 600 --opt rmsprop --batch-size 128 --lr 0.064\ 
--wd 0.00001 --lr-step-size 2 --lr-gamma 0.973 --auto-augment imagenet --random-erase 0.2

Submitted batch job 34241491

Then we took the 3 last checkpoints (epochs 549, 528, 408) that improved the Acc@1 and averaged their parameters using the following script:

# from https://github.com/pytorch/fairseq/blob/master/scripts/average_checkpoints.py
import collections
import torch


def average_checkpoints(inputs):
    params_dict = collections.OrderedDict()
    params_keys = None
    new_state = None
    num_models = len(inputs)
    for fpath in inputs:
        with open(fpath, "rb") as f:
            state = torch.load(
                f,
                map_location=(
                    lambda s, _: torch.serialization.default_restore_location(s, "cpu")
                ),
            )
        # Copies over the settings from the first checkpoint
        if new_state is None:
            new_state = state
        model_params = state["model"]
        model_params_keys = list(model_params.keys())
        if params_keys is None:
            params_keys = model_params_keys
        elif params_keys != model_params_keys:
            raise KeyError(
                "For checkpoint {}, expected list of params: {}, "
                "but found: {}".format(f, params_keys, model_params_keys)
            )
        for k in params_keys:
            p = model_params[k]
            if isinstance(p, torch.HalfTensor):
                p = p.float()
            if k not in params_dict:
                params_dict[k] = p.clone()
                # NOTE: clone() is needed in case of p is a shared parameter
            else:
                params_dict[k] += p
    averaged_params = collections.OrderedDict()
    for k, v in params_dict.items():
        averaged_params[k] = v
        if averaged_params[k].is_floating_point():
            averaged_params[k].div_(num_models)
        else:
            averaged_params[k] //= num_models
    new_state["model"] = averaged_params
    return new_state


def avg(epochs, filename):
    paths = ["model_{}.pth".format(i) for i in epochs]
    weights = average_checkpoints(paths)
    torch.save(weights, filename.format(len(epochs)))

avg([549, 528, 408], "model_best{}avg.pth")

Validated with:

python -m torch.distributed.launch --nproc_per_node=8 --use_env train.py\
 --model mobilenet_v3_large --test-only --pretrained

Submitted batch job 34643680

Accuracy metrics:
Acc@1 74.042 Acc@5 91.340

Speed Benchmark: 0.0411 sec per image on CPU

[codecov]

Codecov Report

Merging #3182 (5030435) into master (4cbe714) will increase coverage by 0.30%.
The diff coverage is 94.95%.

@@            Coverage Diff             @@
##           master    #3182      +/-   ##
==========================================
+ Coverage   73.49%   73.79%   +0.30%     
==========================================
  Files         101      102       +1     
  Lines        9235     9354     +119     
  Branches     1477     1490      +13     
==========================================
+ Hits         6787     6903     +116     
- Misses       1991     1993       +2     
- Partials      457      458       +1     

Impacted Files	Coverage Δ
torchvision/models/mobilenetv3.py	94.91% <94.91%> (ø)
torchvision/models/mobilenet.py	100.00% <100.00%> (ø)
torchvision/models/mobilenetv2.py	86.51% <0.00%> (+3.37%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4cbe714...9a758a8. Read the comment docs.

[vfdev-5]

Nice PR @datumbox !
Few nits...

[datumbox]

The failing builds seem unrelated. See issue #3183

[datumbox]

I added a few comments to assist review.

[datumbox]

Pending S3 bucket access and training finalization.

[datumbox]

These hardcoded params are crucial for convergence =. They can be turned into args.

[datumbox]

I will merge this PR on a separate branch to continue with the changes necessary for Object Detection. I'll send a new PR on master once all changes are final.