Following papers are implemented using PyTorch.
- ResNet (1512.03385)
- ResNet-preact (1603.05027)
- WRN (1605.07146)
- DenseNet (1608.06993)
- PyramidNet (1610.02915)
- ResNeXt (1611.05431)
- shake-shake (1705.07485)
- Cutout (1708.04552)
- Random Erasing (1708.04896)
- SENet (1709.01507)
- Mixup (1710.09412)
- Python 3.6.3
- PyTorch
- torchvision
- tensorboardX (optional)
$ ./main.py --arch resnet_preact --depth 56 --outdir results
$ ./main.py --arch resnet_preact --depth 56 --outdir results --use_cutout
$ ./main.py --arch resnet_preact --depth 56 --outdir results --use_random_erasing
$ ./main.py --arch resnet_preact --depth 56 --outdir results --use_mixup
$ ./main.py --arch wrn --outdir results --scheduler cosine
| Model | Test Error (median of 3 runs) | Test Error (in paper) | Training Time |
|---|---|---|---|
| VGG-like (depth 15, w/ BN, channel 64) | 7.29 | N/A | 1h20m |
| ResNet-110 | 6.52 | 6.43 (best), 6.61 ± 0.16 | 3h06m |
| ResNet-preact-110 | 6.47 | 6.37 (median of 5 runs) | 3h05m |
| ResNet-preact-164 bottleneck | 5.90 | 5.46 (median of 5 runs) | 4h01m |
| ResNet-preact-1001 bottleneck | 4.62 (median of 5 runs), 4.69 ± 0.20 | ||
| WRN-28-10 | 4.03 | 4.00 (median of 5 runs) | 16h10m |
| WRN-28-10 w/ dropout | 3.89 (median of 5 runs) | ||
| DenseNet-100 (k=12) | 3.87 (1 run) | 4.10 (1 run) | 24h28m* |
| DenseNet-100 (k=24) | 3.74 (1 run) | ||
| DenseNet-BC-100 (k=12) | 4.69 | 4.51 (1 run) | 15h20m |
| DenseNet-BC-250 (k=24) | 3.62 (1 run) | ||
| DenseNet-BC-190 (k=40) | 3.46 (1 run) | ||
| PyramidNet-110 (alpha=84) | 4.40 | 4.26 ± 0.23 | 11h40m |
| PyramidNet-110 (alpha=270) | 3.92 (1 run) | 3.73 ± 0.04 | 24h12m* |
| PyramidNet-164 bottleneck (alpha=270) | 3.44 (1 run) | 3.48 ± 0.20 | 32h37m* |
| PyramidNet-272 bottleneck (alpha=200) | 3.31 ± 0.08 | ||
| ResNeXt-29 4x64d | 3.89 | ~3.75 (from Figure 7) | 31h17m |
| ResNeXt-29 8x64d | 3.97 (1 run) | 3.65 (average of 10 runs) | 42h50m* |
| ResNeXt-29 16x64d | 3.58 (average of 10 runs) | ||
| shake-shake-26 2x32d (S-S-I) | 3.68 | 3.55 (average of 3 runs) | 33h49m |
| shake-shake-26 2x64d (S-S-I) | 2.88 (1 run) | 2.98 (average of 3 runs) | 78h48m |
| shake-shake-26 2x96d (S-S-I) | 2.90 (1 run) | 2.86 (average of 5 runs) | 101h32m* |
- Differences with papers in training settings:
- Trained WRN-28-10 with batch size 64 (128 in paper).
- Trained DenseNet-BC-100 (k=12) with batch size 32 and initial learning rate 0.05 (batch size 64 and initial learning rate 0.1 in paper).
- Trained ResNeXt-29 4x64d with a single GPU, batch size 32 and initial learning rate 0.025 (8 GPUs, batch size 128 and initial learning rate 0.1 in paper).
- Trained shake-shake models with a single GPU (2 GPUs in paper).
- Trained shake-shake 26 2x64d (S-S-I) with batch size 64, and initial learning rate 0.1.
- Test errors reported above are the ones at last epoch.
- Experiments with only 1 run are done on different computer from the one used for experiments with 3 runs.
$ python -u main.py --arch vgg --seed 7 --outdir results/vgg_15_BN_64/00
$ python -u main.py --arch resnet --depth 110 --block_type basic --seed 7 --outdir results/resnet_basic_110/00
$ python -u main.py --arch resnet_preact --depth 110 --block_type basic --seed 7 --outdir results/resnet_preact_basic_110/00
$ python -u main.py --arch resnet_preact --depth 164 --block_type bottleneck --seed 7 --outdir results/resnet_preact_bottleneck_164/00
$ python -u main.py --arch wrn --depth 28 --widening_factor 10 --seed 7 --outdir results/wrn_28_10/00
$ python -u main.py --arch densenet --depth 100 --block_type bottleneck --growth_rate 12 --compression_rate 0.5 --batch_size 32 --base_lr 0.05 --seed 7 --outdir results/densenet_BC_100_12/00
$ python -u main.py --arch pyramidnet --depth 110 --block_type basic --pyramid_alpha 84 --seed 7 --outdir results/pyramidnet_basic_110_84/00
$ python -u main.py --arch pyramidnet --depth 110 --block_type basic --pyramid_alpha 270 --seed 7 --outdir results/pyramidnet_basic_110_270/00
$ python -u main.py --arch resnext --depth 29 --cardinality 4 --base_channels 64 --batch_size 32 --base_lr 0.025 --seed 7 --outdir results/resnext_29_4x64d/00
$ python -u main.py --arch resnext --depth 29 --cardinality 8 --base_channels 64 --batch_size 64 --base_lr 0.05 --seed 7 --outdir results/resnext_29_8x64d/00
$ python -u main.py --arch shake_shake --depth 26 --base_channels 32 --shake_forward True --shake_backward True --shake_image True --seed 7 --outdir results/shake_shake_26_2x32d_SSI/00
$ python -u main.py --arch shake_shake --depth 26 --base_channels 64 --shake_forward True --shake_backward True --shake_image True --batch_size 64 --base_lr 0.1 --seed 7 --outdir results/shake_shake_26_2x64d_SSI/00
$ python -u main.py --arch shake_shake --depth 26 --base_channels 96 --shake_forward True --shake_backward True --shake_image True --seed 7 --outdir results/shake_shake_26_2x96d_SSI/00
| Model | Test Error (1 run) | # of Epochs | Training Time |
|---|---|---|---|
| WRN-28-10, Cutout 16 | 3.19 | 200 | 16h23m |
| shake-shake-26 2x64d, Cutout 16 | 2.64 | 1800 | 78h55m |
python -u main.py --arch wrn --depth 28 --outdir results/wrn_28_10_cutout16 --epochs 200 --scheduler cosine --base_lr 0.1 --batch_size 64 --seed 17 --use_cutout --cutout_size 16
python -u main.py --arch shake_shake --depth 26 --base_channels 64 --outdir results/shake_shake_26_2x64d_SSI_cutout16 --epochs 300 --scheduler cosine --base_lr 0.1 --batch_size 64 --seed 17 --use_cutout --cutout_size 16
| Model | Test Error (1 run) | # of Epochs | Training Time |
|---|---|---|---|
| ResNet-preact-20, widening factor 4, Cutout 12 | 4.30 | 160 | 2h37m |
| ResNet-preact-20, widening factor 4, Cutout 14 | 4.27 | 160 | 2h41m |
| ResNet-preact-50, Cutout 12 | 4.56 | 160 | 1h28m |
| ResNet-preact-50, Cutout 14 | 4.34 | 160 | 1h28m |
| ResNet-preact-50, widening factor 4, Cutout 12 | 4.07 | 200 | 8h14m |
| ResNet-preact-50, widening factor 4, Cutout 14 | 4.29 | 200 | 8h14m |
| shake-shake-26 2x32d (S-S-I), Cutout 12 | 4.05 | 300 | 5h32m |
| shake-shake-26 2x32d (S-S-I), Cutout 14 | 4.19 | 300 | 5h24m |
| shake-shake-26 2x96d (S-S-I), Cutout 12 | 4.06 | 300 | 23h54m |
| shake-shake-26 2x96d (S-S-I), Cutout 14 | 3.91 | 300 | 23h58m |
| Model | Test Error (median of 3 runs) | # of Epochs | Training Time |
|---|---|---|---|
| ResNet-preact-20 | 5.14 | 160 | 38m |
| ResNet-preact-20, Cutout 6 | 4.82 | 160 | 38m |
| ResNet-preact-20, Cutout 8 | 4.72 | 160 | 38m |
| ResNet-preact-20, Cutout 10 | 4.88 | 160 | 38m |
| ResNet-preact-20, Cutout 12 | 4.65 | 160 | 38m |
| ResNet-preact-20, Cutout 14 | 4.68 | 160 | 38m |
| ResNet-preact-20, Cutout 16 | 4.84 | 160 | 38m |
| ResNet-preact-20, RandomErasing | 4.77 | 160 | 38m |
| ResNet-preact-20, Mixup | 4.86 | 160 | 49m |
| ResNet-preact-20, Cutout 12 | 4.74 (1 run) | 300 | 1h11m |
| ResNet-preact-20, RandomErasing | 4.70 (1 run) | 300 | 1h11m |
| ResNet-preact-20, Mixup | 4.53 | 300 | 1h31m |
| ResNet-preact-20, Mixup | 4.81 (1 run) | 600 | 3h03m |
- Results reported in the tables are the test errors at last epochs.
- All models are trained using cosine annealing with initial learning rate 0.2.
- Following data augmentations are applied to the training data:
- Images are padded with 4 pixels on each side, and 28x28 patches are randomly cropped from the padded images.
- Images are randomly flipped horizontally.
| Model | Test Error (median of 3 runs) | # of Epochs | Training Time |
|---|---|---|---|
| ResNet-preact-20 | 0.38 | 40 | 9m |
| ResNet-preact-20, Cutout 6 | 0.40 | 40 | 9m |
| ResNet-preact-20, Cutout 8 | 0.32 | 40 | 9m |
| ResNet-preact-20, Cutout 10 | 0.34 | 40 | 9m |
| ResNet-preact-20, Cutout 12 | 0.30 | 40 | 9m |
| ResNet-preact-20, Cutout 14 | 0.34 | 40 | 9m |
| ResNet-preact-20, Cutout 16 | 0.35 | 40 | 9m |
| ResNet-preact-20, RandomErasing | 0.36 | 40 | 9m |
| ResNet-preact-20, Mixup (alpha=1) | 0.39 | 40 | 11m |
| ResNet-preact-20, Mixup (alpha=1) | 0.37 | 80 | 21m |
| ResNet-preact-20, Mixup (alpha=0.5) | 0.33 | 40 | 11m |
| ResNet-preact-20, Mixup (alpha=0.5) | 0.38 | 80 | 21m |
| ResNet-preact-20, widening factor 4, Cutout 12 | 0.29 | 40 | 40m |
| ResNet-preact-50 | 0.39 | 40 | 22m |
| ResNet-preact-50, Cutout 12 | 0.31 | 40 | 22m |
| ResNet-preact-50, widening factor 4, Cutout 12 | 0.29 (1 run) | 40 | 1h40m |
| shake-shake-26 2x32d (S-S-I), Cutout 12 | 0.29 | 100 | 1h48m |
- Results reported in the table are the test errors at last epochs.
- All models are trained using cosine annealing with initial learning rate 0.2.
In this experiment, the effects of the following on classification accuracy are investigated:
- PyramidNet-like residual units
- Cosine annealing of learning rate
- Cutout
- Random Erasing
- Mixup
- Preactivation of shortcuts after downsampling
ResNet-preact-56 is trained on CIFAR-10 with initial learning rate 0.2 in this experiment.
- PyramidNet paper (1610.02915) showed that removing first ReLU in residual units and adding BN after last convolutions in residual units both improve classification accuracy.
- SGDR paper (1608.03983) showed cosine annealing improves classification accuracy even without restarting.
- PyramidNet-like units works.
- It might be better not to preactivate shortcuts after downsampling when using PyramidNet-like units.
- Cosine annealing slightly improves accuracy.
- Cutout, RandomErasing, and Mixup all work great.
- Mixup needs longer training.
| Model | Test Error (median of 5 runs) | Training Time |
|---|---|---|
| w/ 1st ReLU, w/o last BN, preactivate shortcut after downsampling | 6.45 | 95 min |
| w/ 1st ReLU, w/o last BN | 6.47 | 95 min |
| w/o 1st ReLU, w/o last BN | 6.14 | 89 min |
| w/ 1st ReLU, w/ last BN | 6.43 | 104 min |
| w/o 1st ReLU, w/ last BN | 5.85 | 98 min |
| w/o 1st ReLU, w/ last BN, preactivate shortcut after downsampling | 6.27 | 98 min |
| w/o 1st ReLU, w/ last BN, Cosine annealing | 5.72 | 98 min |
| w/o 1st ReLU, w/ last BN, Cutout | 4.96 | 98 min |
| w/o 1st ReLU, w/ last BN, RandomErasing | 5.22 | 98 min |
| w/o 1st ReLU, w/ last BN, Mixup (300 epochs) | 5.11 | 191 min |
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, true, true]' --remove_first_relu false --add_last_bn false --seed 7 --outdir results/experiments/00_preact_after_downsampling/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu false --add_last_bn false --seed 7 --outdir results/experiments/01_w_relu_wo_bn/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn false --seed 7 --outdir results/experiments/02_wo_relu_wo_bn/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu false --add_last_bn true --seed 7 --outdir results/experiments/03_w_relu_w_bn/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn true --seed 7 --outdir results/experiments/04_wo_relu_w_bn/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, true, true]' --remove_first_relu true --add_last_bn true --seed 7 --outdir results/experiments/05_preact_after_downsampling/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn true --scheduler cosine --seed 7 --outdir results/experiments/06_cosine_annealing/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn true --use_cutout --seed 7 --outdir results/experiments/07_cutout/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn true --use_random_erasing --seed 7 --outdir results/experiments/08_random_erasing/00
$ python -u main.py --arch resnet_preact --depth 56 --block_type basic --base_lr 0.2 --preact_stage '[true, false, false]' --remove_first_relu true --add_last_bn true --use_mixup --seed 7 --outdir results/experiments/09_mixup/00
- He, Kaiming, et al. "Deep residual learning for image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016. arXiv:1512.03385
- He, Kaiming, et al. "Identity mappings in deep residual networks." European Conference on Computer Vision. Springer International Publishing, 2016. arXiv:1603.05027, Torch implementation
- Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks. In Richard C. Wilson, Edwin R. Hancock and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016. arXiv:1605.07146, Torch implementation
- Loshchilov, Ilya, and Frank Hutter. "Sgdr: Stochastic gradient descent with warm restarts." In International Conference on Learning Representations, 2017. arXiv:1608.03983, Lasagne implementation
- Huang, Gao, et al. "Densely connected convolutional networks." The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 4700-4708. arXiv:1608.06993, Torch implementation
- Han, Dongyoon, Jiwhan Kim, and Junmo Kim. "Deep pyramidal residual networks." The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5927-5935. arXiv:1610.02915, Torch implementation, Caffe implementation, PyTorch implementation
- Xie, Saining, et al. "Aggregated residual transformations for deep neural networks." The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1492-1500. arXiv:1611.05431, Torch implementation
- Gastaldi, Xavier. "Shake-Shake regularization." In International Conference on Learning Representations, 2017. arXiv:1705.07485, Torch implementation
- DeVries, Terrance, and Graham W. Taylor. "Improved regularization of convolutional neural networks with cutout." arXiv preprint arXiv:1708.04552 (2017). arXiv:1708.04552, PyTorch implementation
- Zhong, Zhun, et al. "Random Erasing Data Augmentation." arXiv preprint arXiv:1708.04896 (2017). arXiv:1708.04896, PyTorch implementation
- Hu, Jie, Li Shen, and Gang Sun. "Squeeze-and-excitation networks." arXiv preprint arXiv:1709.01507 (2017). arXiv:1709.01507
- Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz. "mixup: Beyond Empirical Risk Minimization." arXiv preprint arXiv:1710.09412 (2017). arXiv:1710.09412