- 1 Introduction
- 2 Code and Installation
- 3 Repository of DNNs in vision tasks
- 4 Demo Video and experiment data
- 5 Project member and contact information
At present, LegoDNN includes six kinds of widely used visual DNN applications, including image classification, semantic segmentation, object detection, action recognition, anomaly detection and pose estimation. The DNNs in all visual applications contain a large number of convolution layers and blocks.
[Cite: Rui Han, Qinglong Zhang, Chi Harold Liu, Guoren Wang, Jian Tang, and Lydia Y. Chen. 2021. LegoDNN: block-grained scaling of deep neural networks for mobile vision. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking (MobiCom '21). Association for Computing Machinery, New York, NY, USA, 406–419. https://doi.org/10.1145/3447993.3483249]
-
Image classification applications distinguish different categories of object from an image. The method first takes an image as input, then extracts the image's feature via convolutional layers, and finally outputs the probability of categories via fully connected layers. Take ResNet18 as example, which is shown in Figure (a), it can be divided into three parts: root, four stages and fully connected layer. Other applications use Resent18 pre-trained by ImageNet to extract image features, and make further modifications on the four stages. The pre-trained ResNet18 is so called Backbone.
-
Semantic segmentation applications are widely used in medical images and driverless scenes. A typical DNN model has an encoder-decoder structure, in which the encoder corresponds to an image classification network and the decoder varies across different DNNs. For example, in fully convolutional networks (FCN)~\cite{long2015fully} (Figure (b)), the encoder corresponds to the four stages in ResNet and the decoder contains four convolution layers.
-
Object detection applications detect coordinates of the frames containing objects (e.g., people, dogs, cars) and recognize the objects. Its mainstream networks can be divided into three parts: Backbone, net and detector. Figure (c) shows a popular object detection network YOLO-V3. Its backbone is a ResNet18 which is divided into two parts :a root convolution layer and four stages here. Its detector is the two conected convolution layers before each output. All the remaining convolution layers form the net.
-
Action recognition applications recognize an object's actions in video clips, such as speaking, waving, etc. As shown in Figure (d), a classical two-stream convolutional networks is presented. The network is divided into spatial convolutional network and temporal convolutional network, both of which use image classification networks to perform classification tasks.
-
Anomaly detection applications detect anomalies in data, particularly the image and video data. This network can be divided into two categories: (1) self-training-based model; (2) GAN-based model. As shown in Figure (e1) and Figure (e2), self-training-based model uses ResNet18 to extract data's feature, use fully connected layer to make prediction; GAN-based model is a simple and symmetric AutoEncoder model.
-
Pose estimation focuses on the problem of identifying the orientation of a 3-D object. It has been widely used in many fields such as robot vision, motion tracking, etc. The mainstream pose estimation networks are mainly divided into two categories. The first one first detects an object from an image, and then detects the key points of the object. Network structure of this category is similar to objection detection's. In contrast, the second one first finds the key points and then groups the points. In this way, it can obtain the detect results. Network structure of the second one is similar to semantic segmentation's.
LegoDNN is a lightweight, block-grained and scalable solution for running multi-DNN wrokloads in mobile vision systems. It extracts the blocks of original models via convolutional layers, generates sparse blocks, and retrains the sparse blocks. By composing these blocks, LegoDNN expands the scaling options of original models. At runtime, it optimizes the block selection process using optimization algorithms. The following figure shows a LegoDNN example of ResNet18. This project is a PyTorch-based implementation of LegoDNN, and allows to convert the deep neural networks in the above six mainstream applications to LegoDNN. With LegoDNN, original models are able to dynamically scale at edge, and adapt to changing device resources.
-
Modular Design
This project decomposes the block extracting, retraining and selecting processes of legodnn into various modules. Users can convert their own custom model to legodnn more conveniently by using these module components.
-
Automatic extraction of blocks
This project has implemented a general block extraction algorithm, supporting the automatic block extraction of the models in image classification, object detection, semantic segmentation, attitude estimation, behavior recognition, and anomaly detection applications.
Architecture of legodnn is split into the offline stage and the online stage.
-
Offline Stage：
- At the offline stage, the
block extratoridentifies the original/uncompressed blocks from a DNN model, and feeds them to thedecendant block generatormodule to produce descendant blocks. Theblock retrainermodule then retrains the descendant blocks. Finally, theblock profilermodule profiles all blocks' accuracies, memory and latency information.
- At the offline stage, the
-
Online Stage：
- At the online stage, the
latency estimatormodule estimates latencies of the blocks at edge devices, then sends these latencies with the accuracy and memory information together to thescaling optimatermodule to optimally select blocks. Finally, theblock swichermodule replaces the corresponding blocks in the model with the selected blocks at runtime.
- At the online stage, the
Module details
- BlockManager(AutoBlockManager, BlockExtractor): this module integrates
block extractor,descendant block generator, andblock switcher. The block extractor is responsible for extracting original blocks from an original model's convolution layers. The descendant block generator is responsible for pruning the original blocks to generate multiple sparsity descendant blocks. The block switcher is responsible for replacing blocks with optimal blocks at run time, where the optimal blocks are selected by optimization algorithms. With the AutoBlockManager, this project has implemented automatic extraction of blocks for various models. - BlockRetrainer(BlockTrainer)：this module is used to retrain descendant models to inprove their accuracies. The retraining takes the intermediate data as training data and the sparse blocks as models; the intermediate data is generated by original models as well as original training data; the sparse blocks are generated by original models. The retraining process is quite fast because it only used the intermediate data, reducing the model computation. Meanwhile, these intermediate data can be used in parallel to train the descendant blocks generated from the same original blocks.
- BlockProfile(ServerBlockProfiler)：this module is used to generate analysis and statistics information of the block size, accuracy, etc. The size of a block is the memory it occupies. Since the accuracy loss of a block is different in different combined models, this module selects k different sizes in profiling.
- LatencyProfile(EdgeBlockProfiler)：this module is used to analyze the latency reduction percentage of the blocks on edge devices. The inference latency is obtained by simulating each block's inference on edge device directly. The latency reduction percentage of each block is calculated by using the following formula: (latency of the original block - latency of the currently derived block)/latency of the original block.
- ScailingOptimizer(OptimalRuntime)：this module is used to update and optimize the blocks in real time. By formalizing the block selection as an integer linear programming optimization problem and resolving it in real time, we can continuously obtain the model that owns the maximal accuracy and satisfies the conditions of specific latency and memory limitation.
Offline stage
- Import components and initialize seed
import os os.environ["CUDA_VISIBLE_DEVICES"] = "0" import sys sys.setrecursionlimit(100000) import torch from legodnn import BlockRetrainer, BlockProfiler, LatencyEstimator, ScalingOptimizer from legodnn.common.utils.gen_series_legodnn_models import gen_series_legodnn_models from legodnn.common.utils.dl.common.env import set_random_seed set_random_seed(0) from legodnn.common.detection.model_topology_extraction import topology_extraction from legodnn.common.manager.block_manager.auto_block_manager import AutoBlockManager from legodnn.common.detection.common_detection_manager_1204_new import CommonDetectionManager from legodnn.common.manager.model_manager.common_model_manager import CommonModelManager from cv_task.datasets.image_classification.cifar_dataloader import CIFAR10Dataloader, CIFAR100Dataloader from cv_task.image_classification.cifar.models import resnet18
- Initialize original model
teacher_model = resnet18(num_classes=100).to(device) teacher_model.load_state_dict(torch.load('data/model/resnet110/resnet18.pth')['net'])
- Extract the blocks automatically, then generate descendant blocks and save the blocks to disk using AutoBlockManager
cv_task = 'image_classification' dataset_name = 'cifar100' model_name = 'resnet18' method = 'legodnn' device = 'cuda' compress_layer_max_ratio = 0.125 model_input_size = (1, 3, 32, 32) block_sparsity = [0.0, 0.2, 0.4, 0.6, 0.8] root_path = os.path.join('results/legodnn', cv_task, model_name+'_'+dataset_name + '_' + str(compress_layer_max_ratio).replace('.', '-')) compressed_blocks_dir_path = root_path + '/compressed' trained_blocks_dir_path = root_path + '/trained' descendant_models_dir_path = root_path + '/descendant' block_training_max_epoch = 65 test_sample_num = 100 checkpoint = 'data/models/resnet18/2021-10-20/22-09-22/resnet18.pth' teacher_model = resnet18(num_classes=100).to(device) teacher_model.load_state_dict(torch.load(checkpoint)['net']) print('\033[1;36m--------------------------------> BUILD LEGODNN GRAPH\033[0m') model_graph = topology_extraction(teacher_model, model_input_size, device=device, mode='unpack') model_graph.print_ordered_node() print('\033[1;36m--------------------------------> START BLOCK DETECTION\033[0m') detection_manager = CommonDetectionManager(model_graph, max_ratio=compress_layer_max_ratio) detection_manager.detection_all_blocks() detection_manager.print_all_blocks() model_manager = CommonModelManager() block_manager = AutoBlockManager(block_sparsity, detection_manager, model_manager)
- Retrain the blocks
print('\033[1;36m--------------------------------> START BLOCK EXTRACTION\033[0m') block_extractor = BlockExtractor(teacher_model, block_manager, compressed_blocks_dir_path, model_input_size, device) block_extractor.extract_all_blocks() print('\033[1;36m--------------------------------> START BLOCK TRAIN\033[0m') train_loader, test_loader = CIFAR100Dataloader() block_trainer = BlockTrainer(teacher_model, block_manager, model_manager, compressed_blocks_dir_path, trained_blocks_dir_path, block_training_max_epoch, train_loader, device=device) block_trainer.train_all_blocks()
- Get the profiles about accuracy and memory of the blocks.
server_block_profiler = ServerBlockProfiler(teacher_model, block_manager, model_manager, trained_blocks_dir_path, test_loader, model_input_size, device) server_block_profiler.profile_all_blocks()
Online stage
- Estimate latency of the block
edge_block_profiler = EdgeBlockProfiler(block_manager, model_manager, trained_blocks_dir_path, test_sample_num, model_input_size, device) edge_block_profiler.profile_all_blocks()
- Select the blocks optimally
optimal_runtime = OptimalRuntime(trained_blocks_dir_path, model_input_size, block_manager, model_manager, device) model_size_min = get_model_size(torch.load(os.path.join(compressed_blocks_dir_path, 'model_frame.pt')))/1024**2 model_size_max = get_model_size(teacher_model)/1024**2 + 1 gen_series_legodnn_models(deadline=100, model_size_search_range=[model_size_min, model_size_max], target_model_num=100, optimal_runtime=optimal_runtime, descendant_models_save_path=descendant_models_dir_path, device=device)
Prerequisites
- Linux and Windows
- Python 3.6+
- PyTorch 1.9+
- CUDA 10.2+
Prepare environment
-
Create a conda virtual environment and activate it.
conda create -n legodnn python=3.6 conda active legodnn -
Install PyTorch and torchvision according the official site
Get install params according to the selection in the official site,and copy them to the terminal.Note: please determine whether the CPU version of pytorch or GPU version is installed. If the CPU version of pytorch is installed, please change the
device ='cuda'in the following code todevice ='cpu' -
Install legodnn
git clone https://github.com/LINC-BIT/legodnn.git pip install -r requirements.txt
-
Docker
Using docker Note that these Docker images do not support GPU
Raspberry pi 4B or Jeston TX2 docker run -it bitlinc/legodnn:aarch64-1.0
Note! You should specify a cbc path for some devices in the init method of online/scaling_optimizer.py,like this:
pulp_solver=pulp.COIN_CMD(path="/usr/bin/cbc",msg=False, gapAbs=0)if your device does not have a cbc command in /usr/bin,you should run apt-get install coinor-cbc to install it.
Image classification
Object detection
| Model Name | Data | Script | |
|---|---|---|---|
| ☑ | Fast R-CNN (NIPS'2015) | PARSCAL VOC 2007 | Demo |
| ☑ | YOLOv3 (CVPR'2018) | PARSCAL VOC 2007 | Demo |
Semantic segmentation
| Model Name | Data | Script | |
|---|---|---|---|
| ☑ | FCN (CVPR'2015) | PARSCAL VOC 2012 | Demo |
| ☑ | DeepLab v3 (ArXiv'2017) | PARSCAL VOC 2012 | Demo |
Anomaly detection
| Model Name | Data | Script | |
|---|---|---|---|
| ☑ | GPND (NIPS'2018) | CLatech256 | Demo |
| ☑ | Self-Training (CVPR'2020) | UCSD-Ped1 | Demo |
Pose estimation
| Model Name | Data | Script | |
|---|---|---|---|
| ☑ | DeepPose (CVPR'2014) | MPII | Demo |
| ☑ | SimpleBaselines2D (ECCV'2018) | MPII | Demo |
Action recognition
| Model Name | Data | Script | |
|---|---|---|---|
| ☑ | TSN (ECCV'2016) | HDMB51 | Demo |
| ☑ | TRN (ECCV'2018) | HDMB51 | Demo |
RunningExample.mp4
The model have particular training need to implement a custom model manager based on AbstractModelManager in package legodnn.common.manager.model_manager.abstract_model_manager.
class AbstractModelManager(abc.ABC):
"""Define all attributes of the model.
"""
@abc.abstractmethod
def forward_to_gen_mid_data(self, model: torch.nn.Module, batch_data: Tuple, device: str):
"""Let model perform an inference on given data.
Args:
model (torch.nn.Module): A PyTorch model.
batch_data (Tuple): A batch of data, typically be `(data, target)`.
device (str): Typically be 'cpu' or 'cuda'.
"""
raise NotImplementedError()
@abc.abstractmethod
def dummy_forward_to_gen_mid_data(self, model: torch.nn.Module, model_input_size: Tuple[int], device: str):
"""Let model perform a dummy inference.
Args:
model (torch.nn.Module): A PyTorch model.
model_input_size (Tuple[int]): Typically be `(1, 3, 32, 32)` or `(1, 3, 224, 224)`.
device (str): Typically be 'cpu' or 'cuda'.
"""
raise NotImplementedError()
@abc.abstractmethod
def get_model_acc(self, model: torch.nn.Module, test_loader: DataLoader, device: str):
"""Get the test accuracy of the model.
Args:
model (torch.nn.Module): A PyTorch model.
test_loader (DataLoader): Test data loader.
device (str): Typically be 'cpu' or 'cuda'.
"""
raise NotImplementedError()
@abc.abstractmethod
def get_model_size(self, model: torch.nn.Module):
"""Get the size of the model file (in byte).
Args:
model (torch.nn.Module): A PyTorch model.
"""
raise NotImplementedError()
@abc.abstractmethod
def get_model_flops_and_param(self, model: torch.nn.Module, model_input_size: Tuple[int]):
"""Get the FLOPs and the number of parameters of the model, return as (FLOPs, param).
Args:
model (torch.nn.Module): A PyTorch model.
model_input_size (Tuple[int]): Typically be `(1, 3, 32, 32)` or `(1, 3, 224, 224)`.
"""
raise NotImplementedError()
@abc.abstractmethod
def get_model_latency(self, model: torch.nn.Module, sample_num: int, model_input_size: Tuple[int], device: str):
"""Get the inference latency of the model.
Args:
model (torch.nn.Module): A PyTorch model.
sample_num (int): How many samples is used in the test.
model_input_size (Tuple[int]): Typically be `(1, 3, 32, 32)` or `(1, 3, 224, 224)`.
device (str): Typically be 'cpu' or 'cuda'.
"""
raise NotImplementedError()LegoDNN.Demo.mp4
| Devices | Models and data | Baselines |
|---|---|---|
| Ubuntu 18.04.4 LTS Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz Quadro RTX8000 |
ResNet18 on CIFAR100 MobileNetV2 on CIFAR100 GPNomaly on Coil100 ResNet18 on UCSD Ped1 Faster-RCNN-ResNet50 on PARSCAL VOC2007 YoloV3-DarkNet53 on PARSCAL VOC2007 FCN-ResNet18 on PARSCAL VOC2012 DeepPose-ResNet18 on MPII TSN-ResNet18 on HDMB51 TRN-ResNet18 on HDMB |
Filter Pruning Low Rank Decomposition Knowledge Distillation NestDNN US-Net FN3-channel OFA |
Rui Han, Qinglong Zhang, Gaofeng Xin, Xinyu Guo, Yuxiao Liu, Chi Harold Liu, Guoren Wang
Rui Han: [email protected]
This project is released under the Apache 2.0 license.
1.0.0 was released in 2021.12.20：
Implement basic functions