Inference issue after convert_float_to_float16

[leqiao-1]

Describe the bug

I tried to use mixed precision on model inception_v2.onnx and vgg19.onnx on GPU machine.
At first, I use convert_float_to_float16_model_path with keep_io_types=False, but the inference became even slower.
Here is my script.

• code for conversion

from onnxconverter_common import convert_float_to_float16_model_path
model = "inception_v2.onnx"
new_onnx_model = convert_float_to_float16_model_path(model, keep_io_types=False)
file_path = "new_inception_v2.onnx"
with open(file_path, 'wb') as f:
    f.write(new_onnx_model.SerializeToString())

• code for inference benchmark

import numpy as np
import time
import onnxruntime as ort
def benchmark(model_path):
    session = ort.InferenceSession(model_path)

    total = 0.0
    runs = 200
    input_dict = {"data_0": np.random.random_sample((1,3,224,224)).astype(np.float32)}

    # Warming up
    for i in range(20):
        _ = session.run([], input_dict)

    for i in range(runs):
        start = time.perf_counter()
        _ = session.run([], input_dict)
        end = (time.perf_counter() - start) * 1000
        total += end
    total /= runs
    print(f"Avg: {total:.4f}ms")

Then I tried convert_float_to_float16_model_path with keep_io_types=True. And this time an error occured.

onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from keep_inception_v2.onnx failed:D:\a\_work\1\s\onnxruntime\core\graph\graph.cc:1128 onnxruntime::Graph::Graph [ONNXRuntimeError] : 1 : FAIL : Tensor element type mismatch. 10 != 1

System information

• OS Platform: tested on both Linux and Windows
• ONNX Runtime version: onnxruntime-gpu with version 1.7.0 and 1.9.0
• Python version: 3.6
• onnx version: 1.10.1
• onnxconverter-common version: 1.8.1

To Reproduce

• Code has been shared before.
• Model con be downloaded here

Thanks !

[xiaowuhu]

@leqiao-1 Hi, is this issue still existing? if yes, I will do investigation. if not, I will close this issue.

[xiaowuhu]

if this still bother you, you can reopen it.