- 1. Introduction
- 2. Quick Start
- 3. Training, Evaluation and Inference
- 4. Model Compression
- 5. SHAS
- 6. Inference Deployment
In the process of document scanning, license shooting and so on, sometimes in order to shoot more clearly, the camera device will be rotated, resulting in photo in different directions. At this time, the standard OCR process cannot cope with these issues well. Using the text image orientation classification technology, the direction of the text image can be predicted and adjusted, so as to improve the accuracy of OCR processing. This case provides a way for users to use PaddleClas PULC (Practical Ultra Lightweight image Classification) to quickly build a lightweight, high-precision, practical classification model of text image orientation. This model can be widely used in OCR processing scenarios of rotating pictures in financial, government and other industries.
The following table lists the relevant indicators of the model. The first two lines means that using SwinTransformer_tiny and MobileNetV3_small_x0_35 as the backbone to training. The third to fifth lines means that the backbone is replaced by PPLCNet, additional use of SSLD pretrained model and additional use of hyperparameters searching strategy.
| Backbone | Top1-Acc(%) | Latency(ms) | Size(M) | Training Strategy |
|---|---|---|---|---|
| SwinTranformer_tiny | 99.12 | 89.65 | 111 | using ImageNet pretrained model |
| MobileNetV3_small_x0_35 | 83.61 | 2.95 | 2.6 | using ImageNet pretrained model |
| PPLCNet_x1_0 | 97.85 | 2.16 | 7.1 | using ImageNet pretrained model |
| PPLCNet_x1_0 | 98.02 | 2.16 | 7.1 | using SSLD pretrained model |
| PPLCNet_x1_0 | 99.06 | 2.16 | 7.1 | using SSLD pretrained model + hyperparameters searching strategy |
It can be seen that high accuracy can be getted when backbone is SwinTranformer_tiny, but the speed is slow. Replacing backbone with the lightweight model MobileNetV3_small_x0_35, the speed can be greatly improved, but the accuracy will be greatly reduced. Replacing backbone with faster backbone PPLCNet_x1_0, the accuracy is higher more 14 percentage points than MobileNetv3_small_x0_35. At the same time, the speed can be more faster. After additional using the SSLD pretrained model, the accuracy can be improved by about 0.17 percentage points without affecting the inference speed. Finally, after additional using the hyperparameters searching strategy, the accuracy can be further improved by 1.04 percentage points. At this point, the accuracy is close to that of SwinTranformer_tiny, but the speed is more faster. The training method and deployment instructions of PULC will be introduced in detail below.
Note:
- The Latency is tested on Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz. The MKLDNN is enabled and the number of threads is 10.
- About PP-LCNet, please refer to PP-LCNet Introduction and PP-LCNet Paper.
- Run the following command to install if CUDA9 or CUDA10 is available.
python3 -m pip install paddlepaddle-gpu -i https://mirror.baidu.com/pypi/simple- Run the following command to install if GPU device is unavailable.
python3 -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simplePlease refer to PaddlePaddle Installation for more information about installation, for examples other versions.
The command of PaddleClas installation as bellow:
pip3 install paddleclasFirst, please click here to download and unzip to get the test demo images.
- Prediction with CLI
paddleclas --model_name=text_image_orientation --infer_imgs=pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpgResults:
>>> result
class_ids: [0, 2], scores: [0.85615, 0.05046], label_names: ['0', '180'], filename: pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg
Predict complete!
Note: If you want to test other images, only need to specify the --infer_imgs argument, and the directory containing images is also supported.
- Prediction in Python
import paddleclas
model = paddleclas.PaddleClas(model_name="text_image_orientation")
result = model.predict(input_data="pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg")
print(next(result))Note: The result returned by model.predict() is a generator, so you need to use the next() function to call it or for loop to loop it. And it will predict with batch_size size batch and return the prediction results when called. The default batch_size is 1, and you also specify the batch_size when instantiating, such as model = paddleclas.PaddleClas(model_name="text_image_orientation", batch_size=2). The result of demo above:
>>> result
[{'class_ids': [0, 2], 'scores': [0.85615, 0.05046], 'label_names': ['0', '180'], 'filename': 'pulc_demo_imgs/text_image_orientation/img_rot0_demo.jpg'}]
Please refer to Installation to get the description about installation.
The model provided in 1 section is trained using internal data, which has not been open source. So we provide a dataset with ICDAR2019-ArT, XFUND and ICDAR2015 to experience.
The data used in this case can be getted by processing the open source data. The detailed processes are as follows:
Considering the resolution of original image is too high to need long training time, all the data are scaled in advance. Keeping image aspect ratio, the short edge is scaled to 384. Then rotate the data clockwise to generate composite data of 90 degrees, 180 degrees and 270 degrees respectively. Among them, 41460 images generated by ICDAR2019-ArT and XFUND are randomly divided into training set and verification set according to the ratio of 9:1. 6000 images generated by ICDAR2015 are used as supplementary data in the experiment of SKL-UGI knowledge distillation.
Some image of the processed dataset is as follows:
And you can also download the data processed directly.
cd path_to_PaddleClas
Enter the dataset/ directory, download and unzip the dataset.
cd dataset
wget https://paddleclas.bj.bcebos.com/data/PULC/text_image_orientation.tar
tar -xf text_image_orientation.tar
cd ../The datas under text_image_orientation directory:
├── img_0
│ ├── img_rot0_0.jpg
│ ├── img_rot0_1.png
...
├── img_90
│ ├── img_rot90_0.jpg
│ ├── img_rot90_1.png
...
├── img_180
│ ├── img_rot180_0.jpg
│ ├── img_rot180_1.png
...
├── img_270
│ ├── img_rot270_0.jpg
│ ├── img_rot270_1.png
...
├── distill_data
│ ├── gt_7060_0.jpg
│ ├── gt_7060_90.jpg
...
├── train_list.txt
├── train_list.txt.debug
├── train_list_for_distill.txt
├── test_list.txt
├── test_list.txt.debug
└── label_list.txt
Where img_0/, img_90/, img_180/ and img_270/ are data of 4 angles respectively. The train_list.txt and val_list.txt are label files of training data and validation data respectively. The file train_list.txt.debug and val_list.txt.debug are subset of train_list.txt and val_list.txt respectively. distill_data/ is the supplementary data, which will be used for SKL-UGI knowledge distillation, and its label file is train_list_for_distill.txt.
Note:
- About the contents format of
train_list.txtandval_list.txt, please refer to Description about Classification Dataset in PaddleClas. - About the
train_list_for_distill.txt, please refer to Knowledge Distillation Label.
The details of training config in ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml. The command about training as follows:
export CUDA_VISIBLE_DEVICES=0,1,2,3
python3 -m paddle.distributed.launch \
--gpus="0,1,2,3" \
tools/train.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yamlThe best metric of validation data is about 0.99.
Note:
- The metric mentioned in this document are training on large-scale internal dataset. When using demo data to train, this metric cannot be achieved because the dataset is small and the distribution is different from large-scale internal data. You can further expand your own data and use the optimization method described in this case to achieve higher accuracy.
After training, you can use the following commands to evaluate the model.
python3 tools/eval.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \
-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"Among the above command, the argument -o Global.pretrained_model="output/PPLCNet_x1_0/best_model" specify the path of the best model weight file. You can specify other path if needed.
After training, you can use the model that trained to infer. Command is as follow:
python3 tools/infer.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \
-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"The results:
[{'class_ids': [0, 2], 'scores': [0.85615, 0.05046], 'file_name': 'deploy/images/PULC/text_image_orientation/img_rot0_demo.jpg', 'label_names': ['0', '180']}]
Note:
- Among the above command, argument
-o Global.pretrained_model="output/PPLCNet_x1_0/best_model"specify the path of the best model weight file. You can specify other path if needed. - The default test image is
deploy/images/PULC/text_image_orientation/img_rot0_demo.jpg. And you can test other image, only need to specify the argument-o Infer.infer_imgs=path_to_test_image. - The Top2 result would be printed.
0means that the text direction of the drawing is 0 degrees,90means that 90 degrees clockwise,180means that 180 degrees clockwise,270means that 270 degrees clockwise.
SKL-UGI is a simple but effective knowledge distillation algrithem proposed by PaddleClas.
Training the teacher model with hyperparameters specified in ppcls/configs/PULC/text_image_orientation/PPLCNet/PPLCNet_x1_0.yaml. The command is as follow:
export CUDA_VISIBLE_DEVICES=0,1,2,3
python3 -m paddle.distributed.launch \
--gpus="0,1,2,3" \
tools/train.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \
-o Arch.name=ResNet101_vdThe best metric of validation data is about 0.996. The best teacher model weight would be saved in file output/ResNet101_vd/best_model.pdparams.
Note: Training ResNet101_vd need more GPU memory. So you can reduce batch_size and learning rate at the same time, such as: -o DataLoader.Train.sampler.batch_size=64, Optimizer.lr.learning_rate=0.1.
The training strategy, specified in training config file ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0_distillation.yaml, the teacher model is ResNet101_vd and the student model is PPLCNet_x1_0.
The command is as follow:
export CUDA_VISIBLE_DEVICES=0,1,2,3
python3 -m paddle.distributed.launch \
--gpus="0,1,2,3" \
tools/train.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0_distillation.yaml \
-o Arch.models.0.Teacher.pretrained=output/ResNet101_vd/best_modelThe best metric is about 0.99. The best student model weight would be saved in file output/DistillationModel/best_model_student.pdparams.
The hyperparameters used by 3.2 section and 4.1 section are according by Hyperparameters Searching in PaddleClas. If you want to get better results on your own dataset, you can refer to Hyperparameters Searching to get better hyperparameters.
Note: This section is optional. Because the search process will take a long time, you can selectively run according to your specific. If not replace the dataset, you can ignore this section.
Paddle Inference is the original Inference Library of the PaddlePaddle, provides high-performance inference for server deployment. And compared with directly based on the pretrained model, Paddle Inference can use tools to accelerate prediction, so as to achieve better inference performance. Please refer to Paddle Inference for more information.
Paddle Inference need Paddle Inference Model to predict. Two process provided to get Paddle Inference Model. If want to use the provided by PaddleClas, you can download directly, click Downloading Inference Model.
The command about exporting Paddle Inference Model is as follow:
python3 tools/export_model.py \
-c ./ppcls/configs/PULC/text_image_orientation/PPLCNet_x1_0.yaml \
-o Global.pretrained_model=output/DistillationModel/best_model_student \
-o Global.save_inference_dir=deploy/models/PPLCNet_x1_0_text_image_orientation_inferAfter running above command, the inference model files would be saved in deploy/models/PPLCNet_x1_0_text_image_orientation_infer, as shown below:
├── PPLCNet_x1_0_text_image_orientation_infer
│ ├── inference.pdiparams
│ ├── inference.pdiparams.info
│ └── inference.pdmodel
Note: The best model is from knowledge distillation training. If knowledge distillation training is not used, the best model would be saved in output/PPLCNet_x1_0/best_model.pdparams.
You can also download directly.
cd deploy/models
# download the inference model and decompression
wget https://paddleclas.bj.bcebos.com/models/PULC/text_image_orientation_infer.tar && tar -xf text_image_orientation_infer.tar
After decompression, the directory models should be shown below.
├── text_image_orientation_infer
│ ├── inference.pdiparams
│ ├── inference.pdiparams.info
│ └── inference.pdmodel
Return the directory deploy:
cd ../
Run the following command to classify text image orientation about image ./images/PULC/text_image_orientation/img_rot0_demo.png.
# Use the following command to predict with GPU.
python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml
# Use the following command to predict with CPU.
python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml -o Global.use_gpu=FalseThe prediction results:
img_rot0_demo.jpg: class id(s): [0, 2], score(s): [0.86, 0.05], label_name(s): ['0', '180']
Among the results, 0 means that the text direction of the drawing is 0 degrees, 90 means that 90 degrees clockwise, 180 means that 180 degrees clockwise, 270 means that 270 degrees clockwise.
If you want to predict images in directory, please specify the argument Global.infer_imgs as directory path by -o Global.infer_imgs. The command is as follow.
# Use the following command to predict with GPU. If want to replace with CPU, you can add argument -o Global.use_gpu=False
python3.7 python/predict_cls.py -c configs/PULC/text_image_orientation/inference_text_image_orientation.yaml -o Global.infer_imgs="./images/PULC/text_image_orientation/"All prediction results will be printed, as shown below.
img_rot0_demo.jpg: class id(s): [0, 2], score(s): [0.86, 0.05], label_name(s): ['0', '180']
img_rot180_demo.jpg: class id(s): [2, 1], score(s): [0.88, 0.04], label_name(s): ['180', '90']
PaddleClas provides an example about how to deploy with C++. Please refer to Deployment with C++.
Paddle Serving is a flexible, high-performance carrier for machine learning models, and supports different protocol, such as RESTful, gRPC, bRPC and so on, which provides different deployment solutions for a variety of heterogeneous hardware and operating system environments. Please refer Paddle Serving for more information.
PaddleClas provides an example about how to deploy as service by Paddle Serving. Please refer to Paddle Serving Deployment.
Paddle-Lite is an open source deep learning framework that designed to make easy to perform inference on mobile, embeded, and IoT devices. Please refer to Paddle-Lite for more information.
PaddleClas provides an example of how to deploy on mobile by Paddle-Lite. Please refer to Paddle-Lite deployment.
Paddle2ONNX support convert Paddle Inference model to ONNX model. And you can deploy with ONNX model on different inference engine, such as TensorRT, OpenVINO, MNN/TNN, NCNN and so on. About Paddle2ONNX details, please refer to Paddle2ONNX.
PaddleClas provides an example of how to convert Paddle Inference model to ONNX model by paddle2onnx toolkit and predict by ONNX model. You can refer to paddle2onnx for deployment details.