Modern Convolutional Neural Network Architectures

Revision of the designs and implementation of Modern Convolutional Neural Network architectures

---

Introduction to Convolutional Neural Networks

Convolutional Neural Networks (ConvNets or CNNs) are a class of neural networks algorithms that are mostly used in visual recognition tasks such as image classification, object detection, and image segmentation. The use of ConvNets in visual recognition is inarguably one of the biggest inventions of decade 2010s in deep learning community.

A standard ConvNet architecture is typically made of 3 main layers that are convolution, max-pooling, and fully connected layers. Convolution layers are the main component of ConvNets. They are used for extracting features in images using filters.

Pooling layers are used for downsampling the activation or feature maps produced by convolutional layers. Downsampling can also be achieved by using strides(greater than 1) in a normal convolution layer, but max-pooling layers don't have any learnable parameters and they introduce translational invariance which improves model generalization on the cost of spatial inductive bias. Fully connected layers are used for classification purpose(matching learned features with their respective labels). In classification settings, the last fully connected layer is typically activated with softmax activation function!

Example of ConvNets architectures that follow the above structure are AlexNet and VGG. Most modern ConvNet architectures go beyond plain stack of convolution, max-pooling and fully connected layers. For example, architectures like ResNet and other alike networks involve residual connections.

As you go through the materials in this repository, you will learn more about those architectures and how they are implemented. For more about ConvNets, check resources section!

ConvNet Architectures

• AlexNet - Deep Convolutional Neural Networks: implementation, paper

• VGG - Very Deep Convolutional Networks for Large Scale Image Recognition: implementation, paper

• GoogLeNet(Inceptionv1) - Going Deeper with Convolutions: implementation, paper

• ResNet - Deep Residual Learning for Image Recognition: implementation, annotated paper paper

• ResNeXt - Aggregated Residual Transformations for Deep Neural Networks: implementation, annotated paper, paper

• Xception - Deep Learning with Depthwise Separable Convolutions: implementation, annotated paper, paper

• DenseNet - Densely Connected Convolutional Neural Networks: implementation, annotated paper, paper

• MobileNetV1 - Efficient Convolutional Neural Networks for Mobile Vision Applications: implementation, annotated paper, paper

• MobileNetV2 - Inverted Residuals and Linear Bottlenecks: implementation annotated paper, paper

• EfficientNet - Rethinking Model Scaling for Convolutional Neural Networks: implementation, annotated paper, paper. See also EfficientNetV2

• RegNet - Designing Network Design Spaces: implementation, annotated paper, paper. See also this

• ConvMixer - Patches are All You Need?: implementation, annotated paper, paper.

• ConvNeXt - A ConvNet for the 2020s: implementation, annotated paper, paper

Choosing a ConvNet Architecture

Computer vision community is blessed with having many vision architectures that work great across many platforms or hardwares. But, having many options means it is not easy to choose an architecture that suits a given problem. How can you choose a ConvNet architecture for your problem?

The first rule of thumb is that you should not try to design your own architecture from scratch. If you are working on generic problem, it never hurts to start with ResNet-50. If you are building a mobile-based visual application where there is limited computation resources, try MobileNets(or other mobile friendly architectures like ShuffleNetv2 or ESPNetv2).

For a better trade-off between accuracy and computation efficiency, try EfficientNet or latest ConvNeXt!

That said, choosing architecture(or learning algorithm) is a no free-lunch scenario. There is no universal architecture. There is no single architecture that is guaranteed to work for all datasets and problems. It's all experimentation. It's all trying!

If you are a visionary or like to stay on the bleeding edge of the field, take a look at vision transformers! We don't know yet, but they might be the successor of ConvNets!

Important Notes

The implementations of ConvNets architectures contained in this repository are not optimized for training but rather to understand how those networks were designed, principal components that makes them and how they evolved overtime. LeNet-5(LeCunn, 1998) had 5 layers(with learnable weights, pooling layers excluded). AlexNet(Krizhevsky et al., 2012) had 8 layers. Few years later, Residual Networks(He et al., 2015) made the trends after showing that it's possible to train networks of over 100 layers. Today, residual networks are still one of the most widely used architecture across a wide range of visual tasks and they impacted the design of language architectures.

Computer vision research community is very vibrant. With the intelligent frameworks and better architectures we have to day, understanding how networks architectures are designed before you can throw them in your dataset is never a neccesity, but it's one of the best ways to stay on top of this vibrant and fast-ever changing field!

If you want to use ConvNets for solving a visual recognition tasks such as image classification or object detection, you can get up running quickly by getting the models (and their pretrained weights) from tools like Keras, TensorFlow Hub, PyTorch Vision, Timm PyTorch Image Models, GluonCV, and OpenMML Lab.

References Implementations

• Keras Applications
• Timm PyTorch Image Models
• PyTorch Vision
• Machine Learning Tokyo

Further Learning

If you would like to learn more about ConvNets/CNNs, below are some few amazing resources:

• CS231n Convolutional Neural Networks for Visual Recognition - Lecture 5 and Lecture 9

• Deep Learning for Computer Vision - Lecture 8

• Paper With Code - Convolutional Neural Networks

• MIT Introduction to Deep Learning - Lecture 3

• CS230 Deep Learning - CNN Cheatsheet

• CNNs Interactive Explainer

Citation

If you use any material in this repository, you are welcome to cite it with:

@misc{Nyandwi2022ModernConvNets,
  author = {Nyandwi, Jean de Dieu},
  title = {ModernConvNets},
  year = {2022},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/Nyandwi/ModernConvNets},
}

---

I had a joy learning, revising, and implementing CNN architectures. While going through the materials in this repository, I hope you will enjoy them as much as I did!

For any error, suggestion, or simply anything, you can reach out through email, Twitter or LinkedIn.

---