Why does ptq quantization yield different results?

[lucy3589]

Same onnx, same check set, same code. Why does fp32_onnx change to trt-int8 every time? Is the mAP measured differently on the verification set?

I use samples/python/efficientdet/build_engine.py. and modify image_bathcer.py-resize_pad .

def resize_pad(image, pad_color=(114,114,114)):
            width, height = image.size
            width_scale = width / self.width
            height_scale = height / self.height
            scale = 1.0 / max(width_scale, height_scale)
            _w,_h=round(width * scale), round(height * scale)
            image = image.resize((_w,_h), resample=Image.BILINEAR)

            dw, dh = self.width - _w, self.height-_h
            dw /= 2  # divide padding into 2 sides
            dh /= 2
            top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))
            left, right = int(round(dw - 0.1)), int(round(dw + 0.1))

            pad = Image.new("RGB", (self.width, self.height))
            pad.paste(pad_color, [0, 0, self.width, self.height])
            pad.paste(image,(left, top))
            # pad.show()
            return pad, scale

shuffle use random.seed(47) trt.IInt8EntropyCalibrator2

python build_engine.py --onnx yolov8n-seg.onnx --engine yolov8n-seg-int8.engine --precision int8 --calib_input datasets\coco-seg\images\sample --calib_cache calibrationall8.cache

[zerollzeng]

Sorry I don't get your problem, do you mean every time you do calibration, you get slightly different calibrationall8.cache?

[lucy3589]

Sorry I don't get your problem, do you mean every time you do calibration, you get slightly different calibrationall8.cache?

Yes. That's what you said

[zerollzeng]

I think it's expected. @ttyio for more information.

[ttyio]

closing since no activity for more than 3 weeks, thanks all