Unified Inference Frontend (UIF) 1.1 User Guide
+Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||||||||||||||||||
| PyTorch | -Supports 1.4 - 1.10 | +Supports 1.7 - 1.13 | Yes | Yes | Yes | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TensorFlow | -Supports 2.3 - 2.8 | +Supports 2.4 - 2.12 | Yes | No | No | @@ -150,13 +150,13 @@ model = pruning_runner.prune(removal_ratio=0.2) ``` Run analysis only once for the same model. You can prune the model iteratively without re-running analysis because there is only one pruned model generated for a specific pruning ratio. -The subnetwork obtained by pruning may not be very good because an approximate algorithm is used to generate this unique pruned model according to the analysis result. +The subnetwork obtained by pruning may not be perfect because an approximate algorithm is used to generate this unique pruned model according to the analysis result. The one-step pruning method can generate a better subnetwork. #### One-step Pruning -The method also includes two stages: adaptive batch normalization (BN) based searching for pruning strategy, and pruned model generation. +The method also includes two stages: adaptive batch normalization (BN) based searching for pruning strategy and pruned model generation. After searching, a file named `.vai/your_model_name.search` is generated in which the search result (pruning strategies and corresponding evaluation scores) is stored. You can get the final pruned model in one-step. `num_subnet` provides the number of candidate subnetworks satisfying the sparsity requirement to be searched. @@ -186,7 +186,7 @@ The one-step pruning method has several advantages over the iterative approach: - The workflow is simpler because you can obtain the final pruned model in one step without iterations. - Retraining a slim model is faster than a sparse model. -There are two disadvantages to one-step pruning: one is that the random generation of pruning strategy is unstable. The other is that the subnetwork searching must be performed once for every pruning ratio. +There are two disadvantages to one-step pruning: one is that the random generation of pruning strategy is unstable, and the other is that the subnetwork searching must be performed once for every pruning ratio. ### Retraining the Pruned Model @@ -323,7 +323,7 @@ The searching result looks like the following: ``` ### Getting a Subnetwork -Call `get_static_subnet()` to get a specific subnetwork. The `static_subnet` can be used for finetuning and doing quantization. +Call `get_static_subnet()` to get a specific subnetwork. The `static_subnet` can be used for finetuning and quantization. ```python pareto_global = ofa_pruner.load_subnet_config('pareto_global.txt') @@ -393,15 +393,15 @@ sparse_model = runner.prune(ratio=0.2) ``` **Note:** `ratio` is only an approximate target value and the actual pruning ratio may not be exactly equal to this value. -The returned model from `prune()` is sparse which means the pruned channels are set to zeros and model size remains unchanged. +The returned model from `prune()` is sparse, which means that the pruned channels are set to zeros and model size remains unchanged. The sparse model has been used in the iterative pruning process. The sparse model is converted to a pruned dense model only after pruning is completed. -Besides returning a sparse model, the pruning runner generates a specification file in the `.vai` directory that describes how each layer will be pruned. +Besides returning a sparse model, the pruning runner generates a specification file in the `.vai` directory that describes how each layer is pruned. ### 4.1.4.4: Fine-tuning a Sparse Model -Training a sparse model is no different from training a normal model. The model will maintain sparsity internally. There is no need for any additional actions other than adjusting the hyper-parameters. +Training a sparse model is no different from training a normal model. The model maintains sparsity internally. There is no need for any additional actions other than adjusting the hyper-parameters. ```python sparse_model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"]) @@ -425,7 +425,7 @@ sparse_model = runner.prune(ratio=0.5) ``` ### 4.1.4.6: Getting the Pruned Model -When the iterative pruning is completed, a sparse model is generated which has the same number of parameters as the original model but with many of them now set to zero. +When the iterative pruning is completed, a sparse model is generated, which has the same number of parameters as the original model but with many of them now set to zero. Call `get_slim_model()` to remove zeroed parameters from the sparse model and retrieve the pruned model: diff --git a/docs/4_deploy_your_own_model/quantize_model/pt_resnet18_quant.py b/docs/4_deploy_your_own_model/quantize_model/pt_resnet18_quant.py index cf7c20b..86ef9b8 100644 --- a/docs/4_deploy_your_own_model/quantize_model/pt_resnet18_quant.py +++ b/docs/4_deploy_your_own_model/quantize_model/pt_resnet18_quant.py @@ -213,11 +213,14 @@ def quantization(title='optimize', inspector.inspect(quant_model, (input,), device=device) sys.exit() else: - ## new api #################################################################################### + # This function call will create a quantizer object and setup it. + # Eager mode model code will be converted to graph model. + # Quantization is not done here if it needs calibration. quantizer = torch_quantizer( quant_mode, model, (input), device=device, quant_config_file=config_file) + # Get the converted model to be quantized. quant_model = quantizer.quant_model ##################################################################################### @@ -252,16 +255,22 @@ def quantization(title='optimize', acc_org5 = 0.0 loss_org = 0.0 - #register_modification_hooks(model_gen, train=False) + # This function call is to do forward loop for model to be quantized. + # Quantization calibration will be done after it. acc1_gen, acc5_gen, loss_gen = evaluate(quant_model, val_loader, loss_fn) - # logging accuracy + # Logging accuracy + # If quant_mode is 'calib', ignore the accuracy log because it is not the final accuracy can be got. + # Only check the accuray if quant_mode is 'test'. print('loss: %g' % (loss_gen)) print('top-1 / top-5 accuracy: %g / %g' % (acc1_gen, acc5_gen)) - # handle quantization result if quant_mode == 'calib': + # Exporting intermediate files will be used when quant_mode is 'test'. This is must. quantizer.export_quant_config() + if quant_mode == 'test': + # Exporint ONNX format quantized model + quantizer.export_onnx_model() if __name__ == '__main__': diff --git a/docs/4_deploy_your_own_model/quantize_model/quantizemodel.md b/docs/4_deploy_your_own_model/quantize_model/quantizemodel.md index c45a1f3..eb2a083 100644 --- a/docs/4_deploy_your_own_model/quantize_model/quantizemodel.md +++ b/docs/4_deploy_your_own_model/quantize_model/quantizemodel.md @@ -1,6 +1,6 @@
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
||
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
Unified Inference Frontend (UIF) 1.1 User Guide+ | Unified Inference Frontend (UIF) 1.2 User Guide |
-It is recommended to set `–input_nodes` to be the last nodes of the preprocessing part and to set *-output_nodes* to be the last nodes of the main graph part because some operations in the pre- and postprocessing parts are not quantizable and might cause errors when compiled by the Vitis AI quantizer if you need to deploy the quantized model to the DPU.
+It is recommended to set `–input_nodes` to be the last nodes of the preprocessing part. Similarly, it is advisable to set *-output_nodes* to be the last nodes of the main graph part because some operations in the pre- and postprocessing parts are not quantizable and might cause errors when compiled by the Vitis AI quantizer if you need to deploy the quantized model to the DPU.
The input nodes might not be the same as the placeholder nodes of the graph. If no in-graph preprocessing part is present in the frozen graph, the placeholder nodes should be set to input nodes.
@@ -321,7 +330,7 @@ After the successful execution of the `vai_q_tensorflow` command, one output fil
|No.|Name|Description|
| :--- | :--- | :--- |
|1|deploy_model.pb|Quantized model for the Vitis AI compiler (extended TensorFlow format) for targeting DPUCZDX8G implementations.|
-|2|quantize_eval_model.pb|Quantized model for evaluation (also, the Vitis AI compiler input for most DPU architectures, like DPUCAHX8H, and DPUCADF8H).|
+|2|quantize_eval_model.pb|Quantized model for evaluation (also, the Vitis AI compiler input for most DPU architectures, like DPUCAHX8H and DPUCADF8H).|
### Evaluating the Quantized Model (Optional)
@@ -368,7 +377,7 @@ Quantization aware training (QAT) is similar to float model training/finetuning,
|No.|Name|Description|
| :--- | :--- | :--- |
-|1|Checkpoint files|Floating-point checkpoint files to start from. Omit this if you training the model from scratch.|
+|1|Checkpoint files|Floating-point checkpoint files to start from. Omit this if you are training the model from scratch.|
|2|Dataset|The training dataset with labels.|
|3|Train Scripts|The Python scripts to run float train/finetuning of the model.|
@@ -400,7 +409,7 @@ optimizer = tf.train.GradientDescentOptimizer()
The `QuantizeConfig` contains the configurations for quantization.
-Some basic configurations such as `input_nodes`, `output_nodes`, `input_shapes` need to be set according to your model structure.
+Some basic configurations such as `input_nodes`, `output_nodes`, and `input_shapes` need to be set according to your model structure.
Other configurations such as `weight_bit`, `activation_bit`, and `method` have default values and can be modified as needed. See [vai_q_tensorflow Usage](#4225-vai_q_tensorflow-usage) for detailed information of all the configurations.
@@ -412,7 +421,7 @@ Other configurations such as `weight_bit`, `activation_bit`, and `method` have d
4. **Evaluate the quantized model and generate the frozen model:** After QAT, generate the frozen model after evaluating the quantized graph with a checkpoint file. This can be done by calling the following function after building the float evaluation graph. As the freeze process depends on the quantize evaluation graph, they are often called together.
- **Note**: Function `decent_q.CreateQuantizeTrainingGraph` and `decent_q.CreateQuantizeEvaluationGraph` will modify the default graph in TensorFlow. They need to be called on different graph phases. `decent_q.CreateQuantizeTrainingGraph` needs to be called on the float training graph while `decent_q.CreateQuantizeEvaluationGraph` needs to be called on the float evaluation graph. `decent_q.CreateQuantizeEvaluationGraph` cannot be called right after calling function `decent_q.CreateQuantizeTrainingGraph`, because the default graph has been converted to a quantize training graph. The correct way is to call it right after the float model creation function.
+ **Note**: Function `decent_q.CreateQuantizeTrainingGraph` and `decent_q.CreateQuantizeEvaluationGraph` modify the default graph in TensorFlow. They need to be called on different graph phases. `decent_q.CreateQuantizeTrainingGraph` needs to be called on the float training graph while `decent_q.CreateQuantizeEvaluationGraph` needs to be called on the float evaluation graph. `decent_q.CreateQuantizeEvaluationGraph` cannot be called right after calling function `decent_q.CreateQuantizeTrainingGraph`, because the default graph has been converted to a quantize training graph. The correct way is to call it right after the float model creation function.
```python
# eval.py
@@ -455,7 +464,7 @@ The suffix contains the iteration information from the checkpoint file and the d
The following are some tips for QAT.
-- **Keras Model:** For Keras models, set `backend.set_learning_phase(1)` before creating the float train graph, and set `backend.set_learning_phase(0)` before creating the float evaluation graph. Moreover, `backend.set_learning_phase()` should be called after `backend.clear_session()`. TensorFlow 1.x QAT APIs are designed for TensorFlow native training APIs. Using Keras `model.fit()` APIs in QAT may lead to some "nodes not executed" issues. It is recommended to use QAT APIs in the TensorFlow 2 quantization tool with Keras APIs.
+- **Keras Model:** For Keras models, set `backend.set_learning_phase(1)` before creating the float train graph, and set `backend.set_learning_phase(0)` before creating the float evaluation graph. Moreover, `backend.set_learning_phase()` should be called after `backend.clear_session()`. TensorFlow 1.x QAT APIs are designed for TensorFlow native training APIs. Using Keras `model.fit()` APIs in QAT might lead to some "nodes not executed" issues. It is recommended to use QAT APIs in the TensorFlow 2 quantization tool with Keras APIs.
- **Dropout**: Experiments show that QAT works better without dropout ops. This tool does not support finetuning with dropouts at the moment and they should be removed or disabled before running QAT. This can be done by setting `is_training=false` when using tf.layers or call `tf.keras.backend.set_learning_phase(0)` when using tf.keras.layers.
@@ -679,7 +688,316 @@ $vai_q_tensorflow dump --input_frozen_graph quantize_results/quantize_eval_model
```
Refer to [Vitis AI Model Zoo](https://github.com/Xilinx/Vitis-AI/tree/master/model_zoo) for more TensorFlow model quantization examples.
-