"No space left on device" issue on auto_convert_mixed_precision_model_path()

[yetingqiaqia]

Hi team,

I found auto_convert_mixed_precision_model_path() is easy to have "No space left on device" issue, where it seems to occupy lots of temporary disk space when running.

Below is an example of failure message, which is failed at running attempt 671:

segments= [*192*, *12*, *6*, *1*, (1), *1*, *3*, (1), *2*, *1*, *1*, (1), *6*, (1), (1), (1), (1), *2*, *1*, *1*, (1), (1), (1), (1), (1), (1), *1*, (1), (1), *1*, (1), (1), 1, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *12*, *3*, 1, 1, 1, *6*, 1, 1, 1, *3*, 1, 1, 1, 1, 1, 1, *3*, 1, 1, 1, 1, *2*, *3*, 1, *2*, *3*, 1, 1, 1, *3*, 1, 1, 1, *3*, 1, *2*, 1, 1, 1, *6*, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *2*, 1, 1, 1, *192*, *48*, 1, 1, 1, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *6*, *6*, *3*, 1, 1, 1, *6*, 1, 1, 1, *3*, 1, 1, 1, *3*, *6*, *24*, *12*, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *12*, *24*, *24*, *24*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *6*, *24*, 1, 1, 1, *3*, *6*, *12*, *24*, 1, 1, 1, *3*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *24*, *48*, *12*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *6*, *6*, *3*, 1, 1, 1, *3*, 1, 1, 1, 1, *2*, *3*, 1, 1, 1, 1, *2*, *3*, 1, 1, 1, *3*, 1, 1, 1, *6*, *3*, 1, 1, 1, 1, *2*, *3*, *3*, *3*, *3*, *3*, *6*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *12*, *3*, 1, 1, 1, *3*, 1, 1, 1, *3*, *3*, 1, 1, 1, *3*, *24*, *24*, 1, 1, 1, 1, 1, 1, 1, *2*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *6*, *6*, *3*, 1, 1, 1, *3*, 1, 1, 1, 1, *2*, *3*, *12*, *12*, *6*, *6*, *6*, *6*, *6*, *6*, 1, *2*, *3*, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, *2*, 1, 1, 1, *13*]
Running attempt 671 excluding conversion of 445 nodes
convert to float 16...
node block list =
['MatMul_211', 'Reshape_216', 'Where_221', 'Reshape_228', 'Softmax_229', 'MatMul_230', 'Transpose_231', 'Reshape_236', 'MatMul_237', 'Add_238', 'Add_239', 'ReduceMean_240', 'Sub_241', 'Pow_243', 'ReduceMean_244', 'Add_246', 'Sqrt_247', 'Mul_249', 'Add_250', 'MatMul_251', 'Add_252', 'Div_255', 'Erf_256', 'Constant_257', 'Add_258', 'Mul_259', 'Constant_260', 'Mul_261', 'MatMul_262', 'Add_263', 'Add_264', 'ReduceMean_265', 'Sub_266', 'Constant_267', 'Pow_268', 'ReduceMean_269', 'Constant_270', 'Add_271', 'Sqrt_272', 'Div_273', 'Mul_274', 'Add_275', 'Slice_291', 'MatMul_292', 'Add_293', 'MatMul_300', 'Add_301', 'Constant_302', 'Slice_306', 'MatMul_307', 'Add_308', 'Constant_309', 'Mul_310', 'Constant_311', 'Unsqueeze_315', 'Concat_316', 'Reshape_317', 'Transpose_318', 'Reshape_324', 'Concat_330', 'Reshape_331', 'Transpose_332', 'Transpose_336', 'MatMul_337', '......', 'Concat_1369', 'Reshape_1370', 'MatMul_1371', 'Add_1372', 'Add_1373', 'ReduceMean_1374', 'Sub_1375', 'ReduceMean_1378', 'Constant_1379', 'Add_1380', 'Sqrt_1381', 'Div_1382', 'Mul_1383', 'Add_1384', 'MatMul_1385', 'Add_1386', 'Cast_1387', 'Constant_1388', 'Div_1389', 'Erf_1390', 'Constant_1391', 'Add_1392', 'Mul_1393', 'Constant_1394', 'Mul_1395', 'MatMul_1396', 'Add_1397', 'Add_1398', 'ReduceMean_1399', 'Sub_1400', 'Constant_1401', 'Pow_1402', 'ReduceMean_1403', 'Constant_1404', 'Add_1405', 'Sqrt_1406', 'Div_1407', 'Mul_1408', 'Add_1409', 'Shape_1410', 'MatMul_1426', 'Add_1427', 'Constant_1428', 'Constant_1432', 'Slice_1433', 'MatMul_1434', 'Add_1435', 'Sub_1501', 'Sqrt_1507', 'Div_1508', 'Mul_1509', 'Add_1510', 'MatMul_1511', 'Add_1512', 'Cast_1513', 'Constant_1514', 'Div_1515', 'Erf_1516', 'Constant_1517', 'Add_1518', 'Mul_1519', 'MatMul_1522', 'Add_1523', 'Add_1524']
Traceback (most recent call last):
  File "fp16_convert.py", line 104, in <module>
    convert_float32_to_mixed_precision_model_path(fp32_model_path, fp16_mixed_model_path, ort_inputs, providers)
  File "fp16_convert.py", line 18, in convert_float32_to_mixed_precision_model_path
    fp32_model_path, input_feed, mixed_precision_model_path, location="graph_mixed_precision_tensor.data", customized_validate_func=validate, keep_io_types=True, provider=providers, verbose=True)
  File "/usr/local/lib/python3.6/site-packages/onnxconverter_common/auto_mixed_precision_model_path.py", line 123, in auto_convert_mixed_precision_model_path
    final_block_list = _find_nodes_blocking_fp16(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/onnxconverter_common/auto_mixed_precision_model_path.py", line 203, in _find_nodes_blocking_fp16
    if _convert_and_check_inference_result(**kwargs):
  File "/usr/local/lib/python3.6/site-packages/onnxconverter_common/auto_mixed_precision_model_path.py", line 244, in _convert_and_check_inference_result
    save_model(model_16, target_model_path, location=location)
  File "/usr/local/lib/python3.6/site-packages/onnxconverter_common/auto_mixed_precision_model_path.py", line 261, in save_model
    onnx.save_model(model, model_path, save_as_external_data=True, location=location)
  File "/usr/local/lib/python3.6/site-packages/onnx/__init__.py", line 200, in save_model
    proto = write_external_data_tensors(proto, basepath)
  File "/usr/local/lib/python3.6/site-packages/onnx/external_data_helper.py", line 263, in write_external_data_tensors
    save_external_data(tensor, filepath)
  File "/usr/local/lib/python3.6/site-packages/onnx/external_data_helper.py", line 180, in save_external_data
    data_file.write(tensor.raw_data)
OSError: [Errno 28] No space left on device

This is my conversion code:

from onnxconverter_common import auto_mixed_precision_model_path
import onnx
import numpy as np
import onnxruntime as ort
from onnxmltools.utils.float16_converter import convert_float_to_float16_model_path
from onnxmltools.utils import save_model


def convert_float32_to_mixed_precision_model_path(fp32_model_path, mixed_precision_model_path, input_feed, providers):
    # Could also use rtol/atol attributes directly instead of this
    def validate(res1, res2):
        for r1, r2 in zip(res1, res2):
            if not np.allclose(r1, r2, rtol=0.01, atol=0.001):
                return False
        return True

    auto_mixed_precision_model_path.auto_convert_mixed_precision_model_path(
        fp32_model_path, input_feed, mixed_precision_model_path, location="graph_mixed_precision_tensor.data", customized_validate_func=validate, keep_io_types=True, provider=providers, verbose=True)
    #onnx.save(model_fp16, mixed_precision_model_path)


def convert_float32_to_float16(fp32_model_path, fp16_model_path):
    from onnxmltools.utils.float16_converter import convert_float_to_float16_model_path
    from onnxmltools.utils import save_model

    new_onnx_model = convert_float_to_float16_model_path(fp32_model_path, keep_io_types=True)
    save_model(new_onnx_model, fp16_model_path)

def convert_float32_to_mixed_precision(fp32_model_path, mixed_precision_model_path, ort_inputs):
    from onnxconverter_common import auto_mixed_precision
    import onnx

    model = onnx.load(fp32_model_path)

    import numpy as np
    np.random.seed(123)

    # Could also use rtol/atol attributes directly instead of this
    def validate(res1, res2):
        for r1, r2 in zip(res1, res2):
            if not np.allclose(r1, r2, rtol=0.01, atol=0.001):
                return False
        return True

    model_fp16 = auto_mixed_precision.auto_convert_mixed_precision(model, ort_inputs, validate, keep_io_types=True)
    onnx.save(model_fp16, mixed_precision_model_path)

def test(onnx_model_path, ort_inputs, ort_output_names):
    import numpy as np
    import time
    np.random.seed(123)
    #Load ort model
    import onnxruntime as ort
    sess_options = ort.SessionOptions()
    sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
    sess_options.intra_op_num_threads = 0

    sess = ort.InferenceSession(onnx_model_path, sess_options, providers=['CUDAExecutionProvider'])
    #sess = ort.InferenceSession(onnx_model_path, sess_options, providers=['CPUExecutionProvider'])

    #warm-up run
    warm_up_start_stamp = time.time()
    onnx_outs = sess.run(ort_output_names, ort_inputs)[0]
    print(f"onnx_outs of warm-up:", onnx_outs)
    print(f"It takes {time.time()-warm_up_start_stamp} to finish warm-up.\n")

    start_stamp = time.time()
    num_batches = 0
    for i in range(num_batches):
        print(f"batch id: {i}")
        onnx_outs = sess.run(ort_output_names, ort_inputs)
        print(f"onnx_outs:", onnx_outs)
        print(f"{i}th batch finished successfully. ")
    print(f"It takes {time.time()-start_stamp} to finish {num_batches} batches.\n")


fp32_model_path = './model/graph.onnx'
#fp16_model_path = './model/8_fp16/graph_fp16.onnx'
#convert_float32_to_float16(fp32_model_path, fp16_model_path)
fp16_mixed_model_path = './model/mixed_precision/graph_mixed_precision.onnx'


ort_inputs={
        'input_ids':[
            [10093,21382,2094,26264,10093,11106,15879,2705,4132,5099,11057,24577,5576,12379,26743,3468,1998,10003,3857,5099,11057,24577,10093,11106,15879,2705,2565,2005,2151,12233,5901,15770,8231,2007,27084,6162,2565,3112,2085,1999,1996,2925,2007,2256,7721,7661,21335,3468,2578,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [2041,5963,21475,4580,5269,2041,5963,5269,2379,2033,4638,2085,2424,2041,5963,21475,4580,5269,2379,2033,2041,5963,21475,4580,12183,14266,1998,5269,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [2270,2636,2005,21904,5937,7523,3784,2270,2636,2156,1015,21904,5937,1055,3042,1016,4769,1017,2287,2062,3046,6575,2085,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [8285,9323,11742,6084,10846,2047,12609,11742,1018,5479,3193,2783,4107,7523,2256,2312,4989,1997,2047,2109,11742,4683,2800,2085,5959,2307,9144,2326,2651,2123,1056,3335,2256,2783,4107,21134,3942,2085,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [2273,2015,3509,16722,4293,2125,2085,4121,5096,2006,2035,2273,2015,3509,16722,2085,9241,3132,3749,3828,2085,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [4268,4542,17764,2039,2000,3438,2125,2131,3201,2005,6172,2039,2000,3438,2125,5096,2012,3137,9746,4497,2085,1998,3828,4497,2085,1998,2131,2039,2000,3438,2125,2012,3137,9746,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [2424,6100,1997,1037,2610,3189,3784,4607,1037,2171,1998,3945,2424,2026,14757,19040,3229,2610,2636,14757,19040,2015,1998,2062,2074,4607,2151,2171,3945,3229,14757,19040,2015,4606,2610,2636,2062,2797,5851,2074,2828,1999,2171,2110,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0],
            [3828,2006,13675,12322,2015,2012,15849,17406,4497,2256,4100,3296,5096,3693,2252,1051,2131,14610,2058,1015,2454,9144,2296,4696,2006,5096,2360,7592,2000,3500,2007,4121,2188,3871,10995,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
            ]
}

ort_output_names = ['seq_embedding']

#print("Test fp32 model: ")
#test(fp32_model_path, ort_inputs, ort_output_names)
#print("f32 model test finished.")

providers=['CUDAExecutionProvider']
print("Convert to mixed precision starts...")
convert_float32_to_mixed_precision_model_path(fp32_model_path, fp16_mixed_model_path, ort_inputs, providers)
print("Conversion finished.")

#print("Test mixed precision model: ")
#test(mixed_precision_model_path, ort_inputs, ort_output_names)
#print("Mixed precision model test finished.")

How to reproduce?
Check this zip file:
https://drive.google.com/file/d/1oJayYH4HCeB6VE1Kp-DiNvqLTMAWsUwc/view?usp=sharing
It includes:

1. code file: fp16_converter.py
2. model file: model/graph.onnx
3. failure log file: prophetNet_conversion.log
   For this error, this log file could help as it contains all the information you need. Otherwise, run the model conversion code needs to take several hours until failure.

Command to run:
python fp16_converter.py

[xiaowuhu]

PR is here: #230