## Usage
-### MobileNetV2 Model Architecture
-
-Refer to the [PTQ Model Usage](PTQ.md#mobilenetv2-model-architecture).
-
-### Helper Functions
-
-Refer to [PTQ Helper Functions](PTQ.md#helper-functions).
-
-### QAT
-
-First, define a training function:
+First, define a training function as below.
+accuracy is in the
```python
-def train_one_epoch(model, criterion, optimizer, data_loader, device, ntrain_batches):
- model.train()
- top1 = AverageMeter('Acc@1', ':6.2f')
- top5 = AverageMeter('Acc@5', ':6.2f')
- avgloss = AverageMeter('Loss', '1.5f')
-
- cnt = 0
- for image, target in data_loader:
- start_time = time.time()
- print('.', end = '')
- cnt += 1
- image, target = image.to(device), target.to(device)
- output = model(image)
- loss = criterion(output, target)
- optimizer.zero_grad()
- loss.backward()
- optimizer.step()
- acc1, acc5 = accuracy(output, target, topk=(1, 5))
- top1.update(acc1[0], image.size(0))
- top5.update(acc5[0], image.size(0))
- avgloss.update(loss, image.size(0))
- if cnt >= ntrain_batches:
- print('Loss', avgloss.avg)
-
- print('Training: * Acc@1 {top1.avg:.3f} Acc@5 {top5.avg:.3f}'
- .format(top1=top1, top5=top5))
- return
-
- print('Full imagenet train set: * Acc@1 {top1.global_avg:.3f} Acc@5 {top5.global_avg:.3f}'
- .format(top1=top1, top5=top5))
- return
+def training_func_for_nc(model):
+ epochs = 8
+ iters = 30
+ optimizer = torch.optim.SGD(model.parameters(), lr=0.0001)
+ for nepoch in range(epochs):
+ model.train()
+ cnt = 0
+ for image, target in train_loader:
+ print('.', end='')
+ cnt += 1
+ output = model(image)
+ loss = criterion(output, target)
+ optimizer.zero_grad()
+ loss.backward()
+ optimizer.step()
+ if cnt >= iters:
+ break
+ if nepoch > 3:
+ # Freeze quantizer parameters
+ model.apply(torch.quantization.disable_observer)
+ if nepoch > 2:
+ # Freeze batch norm mean and variance estimates
+ model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)
+ return model
```
-Fuse modules as PTQ:
+Fuse modules:
```python
model.fuse_model()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.0001)
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
```
-Finally, prepare_qat performs the "fake quantization", preparing the model for quantization-aware training:
+Finally, prepare_qat performs the "fake quantization", preparing the model for quantization-aware training, this function already be implemented as a hook :
```python
torch.quantization.prepare_qat(model, inplace=True)
```
-Training a quantized model with high accuracy requires accurate modeling of numerics at inference. For quantization-aware training, therefore, modify the training loop by doing the following:
-
+Training a quantized model with high accuracy requires accurate modeling of numerics at inference. INC does the training loop by following:
* Switch batch norm to use running mean and variance towards the end of training to better match inference numerics.
* Freeze the quantizer parameters (scale and zero-point) and fine tune the weights.
+
```python
num_train_batches = 20
# Train and check accuracy after each epoch
@@ -88,6 +69,20 @@ for nepoch in range(8):
print('Epoch %d :Evaluation accuracy on %d images, %2.2f'%(nepoch, num_eval_batches * eval_batch_size, top1.avg))
```
+When using QAT in INC, you just need to use these APIs:
+```python
+from neural_compressor.experimental import Quantization, common
+quantizer = Quantization("./conf.yaml")
+quantizer.model = common.Model(model)
+quantizer.q_func = training_func_for_nc
+quantizer.eval_dataloader = val_loader
+q_model = quantizer.fit()
+```
+
+The quantizer.fit() function will return a best quantized model during timeout constrain.
++The yaml define example: [The yaml example](/examples/pytorch/image_recognition/torchvision_models/quantization/qat/fx) + Here, we just perform quantization-aware training for a small number of epochs. Nevertheless, quantization-aware training yields an accuracy of over 71% on the entire imagenet dataset, which is close to the floating point accuracy of 71.9%. More on quantization-aware training: @@ -96,10 +91,6 @@ More on quantization-aware training: * We can simulate the accuracy of a quantized model in floating points since we are using fake-quantization to model the numerics of actual quantized arithmetic. * We can easily mimic post-training quantization. -Intel® Neural Compressor can support QAT calibration for -PyTorch models. Refer to the [QAT model](https://github.com/intel/neural-compressor/tree/master/examples/pytorch/eager/image_recognition/imagenet/cpu/qat/README.md) for step-by-step tuning. - -### Example -View a [QAT example of PyTorch resnet50](/examples/pytorch/image_recognition/torchvision_models/quantization/qat). - +### Examples +For related examples, please refer to the [QAT models](/examples/README.md). diff --git a/docs/Quantization.md b/docs/Quantization.md index 18619185e3f..dd3dbce3c7d 100644 --- a/docs/Quantization.md +++ b/docs/Quantization.md @@ -1,15 +1,77 @@ -Quantization -============ +# Quantization -Quantization refers to processes that enable lower precision inference and training by performing computations at fixed point integers that are lower than floating points. This often leads to smaller model sizes and faster inference time. Quantization is particularly useful in deep learning inference and training, where moving data more quickly and reducing bandwidth bottlenecks is optimal. Intel is actively working on techniques that use lower numerical precision by using training with 16-bit multipliers and inference with 8-bit or 16-bit multipliers. Refer to the Intel article on [lower numerical precision inference and training in deep learning](https://software.intel.com/content/www/us/en/develop/articles/lower-numerical-precision-deep-learning-inference-and-training.html). +Quantization is a widely-used model compression technique that can reduce model size while also improving inference and training latency. +The full precision data converts to low-precision, there is little degradation in model accuracy, but the inference performance of quantized model can gain higher performance by saving the memory bandwidth and accelerating computations with low precision instructions. Intel provided several lower precision instructions (ex: 8-bit or 16-bit multipliers), both training and inference can get benefits from them. +Refer to the Intel article on [lower numerical precision inference and training in deep learning](https://software.intel.com/content/www/us/en/develop/articles/lower-numerical-precision-deep-learning-inference-and-training.html). -Quantization methods include the following three classes: +## Quantization Support Matrix -* [Post-Training Quantization (PTQ)](./PTQ.md) -* [Quantization-Aware Training (QAT)](./QAT.md) -* [Dynamic Quantization](./dynamic_quantization.md) +Quantization methods include the following three types: +
| Types | +Quantization | +Dataset Requirements | +Framework | +Backend | +
|---|---|---|---|---|
| Post-Training Static Quantization (PTQ) | +weights and activations | +calibration | +PyTorch | +PyTorch Eager/PyTorch FX/IPEX | +
| TensorFlow | +TensorFlow/Intel TensorFlow | +|||
| ONNX Runtime | +QLinearops/QDQ | +|||
| Post-Training Dynamic Quantization | +weights | +none | +PyTorch | +PyTorch eager mode/PyTorch fx mode/IPEX | +
| ONNX Runtime | +QIntegerops | +|||
| Quantization-aware Training (QAT) | +weights and activations | +fine-tuning | +PyTorch | +PyTorch eager mode/PyTorch fx mode/IPEX | +
| TensorFlow | +TensorFlow/Intel TensorFlow | +
+
-> **Note** -> -> Dynamic Quantization currently only supports the onnxruntime backend. +### [Post-Training Static Quantization](./PTQ.md) performs quantization on already trained models, it requires an additional pass over the dataset to work, only activations do calibration. +
++ +### [Post-Training Dynamic Quantization](./dynamic_quantization.md) simply multiplies input values by a scaling factor, then rounds the result to the nearest, it determines the scale factor for activations dynamically based on the data range observed at runtime. Weights are quantized ahead of time but the activations are dynamically quantized during inference. +
++ +### [Quantization-aware Training (QAT)](./QAT.md) quantizes models during training and typically provides higher accuracy comparing with post-training quantization, but QAT may require additional hyper-parameter tuning and it may take more time to deployment. +
+
+## Examples of Quantization
+
+For Quantization related examples, please refer to [Quantization examples](/examples/README.md)
diff --git a/docs/infrastructure.md b/docs/design.md
similarity index 96%
rename from docs/infrastructure.md
rename to docs/design.md
index 6b23be3c10c..bee2fa124b8 100644
--- a/docs/infrastructure.md
+++ b/docs/design.md
@@ -1,4 +1,4 @@
-Infrastructure
+Design
=====
Intel® Neural Compressor features an architecture and workflow that aids in increasing performance and faster deployments across infrastructures.
diff --git a/docs/imgs/PTQ.png b/docs/imgs/PTQ.png
new file mode 100644
index 00000000000..9d6d8183409
Binary files /dev/null and b/docs/imgs/PTQ.png differ
diff --git a/docs/imgs/QAT.png b/docs/imgs/QAT.png
new file mode 100644
index 00000000000..27c72efa583
Binary files /dev/null and b/docs/imgs/QAT.png differ
diff --git a/docs/imgs/dynamic_quantization.png b/docs/imgs/dynamic_quantization.png
new file mode 100644
index 00000000000..2a71c0ea1ea
Binary files /dev/null and b/docs/imgs/dynamic_quantization.png differ
diff --git a/docs/imgs/fake_quant.png b/docs/imgs/fake_quant.png
new file mode 100644
index 00000000000..855297a685d
Binary files /dev/null and b/docs/imgs/fake_quant.png differ
diff --git a/docs/orchestration.md b/docs/orchestration.md
new file mode 100755
index 00000000000..945c37e0d28
--- /dev/null
+++ b/docs/orchestration.md
@@ -0,0 +1,57 @@
+Optimization Orchestration
+============
+
+## Introduction
+
+Intel Neural Compressor supports arbitrary meaningful combinations of supported optimization methods under one-shot or multi-shot, such as pruning during quantization-aware training, or pruning and then post-training quantization,
+pruning and then distillation and then quantization.
+
+## Validated Orchestration Types
+
+### One-shot
+
+- Pruning during quantization-aware training
+- Distillation with pattern lock pruning
+- Distillation with pattern lock pruning and quantization-aware training
+
+### Multi-shot
+
+- Pruning and then post-training quantization
+- Distillation and then post-training quantization
+
+## Orchestration user facing API
+
+Neural Compressor defines `Scheduler` class to automatically pipeline execute model optimization with one shot or multiple shots way.
+
+User instantiates model optimization components, such as quantization, pruning, distillation, separately. After that, user could append
+those separate optimization objects into scheduler's pipeline, the scheduler API executes them one by one.
+
+In following example it executes the pruning and then post-training quantization with two-shot way.
+
+```python
+from neural_compressor.experimental import Quantization, Pruning, Scheduler
+prune = Pruning(prune_conf)
+quantizer = Quantization(post_training_quantization_conf)
+scheduler = Scheduler()
+scheduler.model = model
+scheduler.append(prune)
+scheduler.append(quantizer)
+opt_model = scheduler.fit()
+```
+
+If user wants to execute the pruning and quantization-aware training with one-shot way, the code is like below.
+
+```python
+from neural_compressor.experimental import Quantization, Pruning, Scheduler
+prune = Pruning(prune_conf)
+quantizer = Quantization(quantization_aware_training_conf)
+scheduler = Scheduler()
+scheduler.model = model
+combination = scheduler.combine(prune, quantizer)
+scheduler.append(combination)
+opt_model = scheduler.fit()
+```
+
+### Examples
+
+For orchestration related examples, please refer to [Orchestration examples](../examples/README.md).
diff --git a/docs/platform_configuration.md b/docs/platform_configuration.md
new file mode 100644
index 00000000000..764216ddc72
--- /dev/null
+++ b/docs/platform_configuration.md
@@ -0,0 +1,66 @@
+### SYSTEM CONFIGURATION
+| System Configuration | Intel Xeon Platinum 8380 Scalable processor |
|---|---|
| Manufacturer | +Intel Corporation | +
| Product Name | +M50CYP2SBSTD | +
| BIOS Version | +SE5C6200.86B.0022.D64.2105220049 | +
| OS | +Ubuntu 20.04.1 LTS | +
| Kernel | +5.4.0-42-generic | +
| Microcode | +0xd0002b1 | +
| CPU Model | +Intel(R) Xeon(R) Platinum 8380 CPU @ 2.30GHz | +
| Base Frequency | +2.3GHZ | +
| Thread(s) per Core | +2 | +
| Core(s) per Socket | +40 | +
| Socket(s) | +2 | +
| Turbo | +Enabled | +
| Power & Perf Policy | +Balanced | +
| Installed | +256GB (16x16GB DDR4 3200MT/s [3200MT/s]) | +
| NIC Summary | +2x Ethernet Controller 10G X550T | +
| Drive Summary | +1x INTEL_SSDSC2KW01 953.9G, +1x CT1000MX500SSD1 931.5G, +1x CT1000MX500SSD1 931.5G + | +
| Pruning Type | +Pruning Granularity | +Pruning Algorithm | +Framework | +
|---|---|---|---|
| Unstructured Pruning | +Element-wise | +Magnitude | +PyTorch, TensorFlow | +
| Pattern Lock | +PyTorch | +||
| Structured Pruning | +Filter/Channel-wise | +Gradient Sensitivity | +PyTorch | +
| Block-wise | +Group Lasso | +PyTorch | +|
| Element-wise | +Pattern Lock | +PyTorch | +