Error when converting yolox to tensorflow lite

[asizdzbest]

1. OS using Ubuntu 18.04

2. OS Architecture e.g. x64

3->8. Using docker pinto0309/openvino2tensorflow:lastest

9. URL of the repository from which the transformed model was taken: https://github.com/Megvii-BaseDetection/YOLOX

10. Issue Details

I pulled your openvino2tensorflow docker:lastest

docker build -t pinto0309/openvino2tensorflow:latest .

Then using your scipts to convert yolox_nano_320x320.onnx to tensorflow lite:

xhost +local: && \
docker run -it --rm \
-v `pwd`:/home/user/workdir \
-v /tmp/.X11-unix/:/tmp/.X11-unix:rw \
--device /dev/video0:/dev/video0:mwr \
--net=host \
-e XDG_RUNTIME_DIR=$XDG_RUNTIME_DIR \
-e DISPLAY=$DISPLAY \
--privileged \
pinto0309/openvino2tensorflow:latest


MODEL=yolox_nano
H=320
W=320
$INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \
--input_model ${MODEL}_${H}x${W}.onnx \
--data_type FP32 \
--output_dir openvino/${MODEL}/${H}x${W}/FP32


MODEL=yolox_nano
H=320
W=320
openvino2tensorflow \
--model_path openvino/${MODEL}/${H}x${W}/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_no_quant_float32_tflite \
--output_weight_quant_tflite \
--output_float16_quant_tflite \
--output_integer_quant_tflite \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8' \
--output_tfjs \
--output_coreml \
--weight_replacement_config weight_replacement_config_${MODEL}.json
mv saved_model saved_model_${MODEL}_${H}x${W}
`

Then it raise this error:
`
ERROR: Dimension 1 in both shapes must be equal, but are 400 and 100. Shapes are [1,400] and [1,100]. for '{{node tf.concat_17/concat}} = ConcatV2[N=3, T=DT_FLOAT, Tidx=DT_INT32](Placeholder, Placeholder_1, Placeholder_2, tf.concat_17/concat/axis)' with input shapes: [1,1600,85], [1,400,85], [1,100,85], [] and with computed input tensors: input[3] = <2>.
ERROR: model_path  : openvino/yolox_nano/320x320/FP32/yolox_nano_320x320.xml
ERROR: weights_path: openvino/yolox_nano/320x320/FP32/yolox_nano_320x320.bin
ERROR: layer_id    : 666
ERROR: input_layer0 layer_id=457: KerasTensor(type_spec=TensorSpec(shape=(1, 1600, 85), dtype=tf.float32, name=None), name='tf.reshape/Reshape:0', description="created by layer 'tf.reshape'")
ERROR: input_layer1 layer_id=561: KerasTensor(type_spec=TensorSpec(shape=(1, 400, 85), dtype=tf.float32, name=None), name='tf.reshape_1/Reshape:0', description="created by layer 'tf.reshape_1'")
ERROR: input_layer2 layer_id=665: KerasTensor(type_spec=TensorSpec(shape=(1, 100, 85), dtype=tf.float32, name=None), name='tf.reshape_2/Reshape:0', description="created by layer 'tf.reshape_2'")
ERROR: The trace log is below.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/array_ops.py", line 1769, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1217, in concat_v2
    return concat_v2_eager_fallback(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1248, in concat_v2_eager_fallback
    _attr_T, values = _execute.args_to_matching_eager(list(values), ctx, [])
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/execute.py", line 273, in args_to_matching_eager
    tensor = ops.convert_to_tensor(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1566, in convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 346, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 271, in constant
    return _constant_impl(value, dtype, shape, name, verify_shape=False,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
    return _constant_eager_impl(ctx, value, dtype, shape, verify_shape)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 308, in _constant_eager_impl
    t = convert_to_eager_tensor(value, ctx, dtype)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/constant_op.py", line 106, in convert_to_eager_tensor
    return ops.EagerTensor(value, ctx.device_name, dtype)
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/keras_tensor.py", line 244, in __array__
    raise TypeError(
TypeError: Cannot convert a symbolic Keras input/output to a numpy array. This error may indicate that you're trying to pass a symbolic value to a NumPy call, which is not supported. Or, you may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1880, in _create_c_op
    c_op = pywrap_tf_session.TF_FinishOperation(op_desc)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Dimension 1 in both shapes must be equal, but are 400 and 100. Shapes are [1,400] and [1,100]. for '{{node tf.concat_17/concat}} = ConcatV2[N=3, T=DT_FLOAT, Tidx=DT_INT32](Placeholder, Placeholder_1, Placeholder_2, tf.concat_17/concat/axis)' with input shapes: [1,1600,85], [1,400,85], [1,100,85], [] and with computed input tensors: input[3] = <2>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/openvino2tensorflow", line 839, in convert
    tf_layers_dict[layer_id] = tf.concat(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py", line 210, in wrapper
    result = dispatch(wrapper, args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py", line 126, in dispatch
    result = dispatcher.handle(op, args, kwargs)
  File "/usr/local/lib/python3.8/dist-packages/keras/layers/core.py", line 1473, in handle
    return TFOpLambda(op)(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 976, in __call__
    return self._functional_construction_call(inputs, args, kwargs,
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 1114, in _functional_construction_call
    outputs = self._keras_tensor_symbolic_call(
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 848, in _keras_tensor_symbolic_call
    return self._infer_output_signature(inputs, args, kwargs, input_masks)
  File "/usr/local/lib/python3.8/dist-packages/keras/engine/base_layer.py", line 888, in _infer_output_signature
    outputs = call_fn(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/keras/layers/core.py", line 1350, in _call_wrapper
    return self._call_wrapper(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/keras/layers/core.py", line 1382, in _call_wrapper
    result = self.function(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper
    return target(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/array_ops.py", line 1769, in concat
    return gen_array_ops.concat_v2(values=values, axis=axis, name=name)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 1227, in concat_v2
    _, _, _op, _outputs = _op_def_library._apply_op_helper(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/op_def_library.py", line 748, in _apply_op_helper
    op = g._create_op_internal(op_type_name, inputs, dtypes=None,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/func_graph.py", line 599, in _create_op_internal
    return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 3561, in _create_op_internal
    ret = Operation(
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 2041, in __init__
    self._c_op = _create_c_op(self._graph, node_def, inputs,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/ops.py", line 1883, in _create_c_op
    raise ValueError(str(e))
ValueError: Dimension 1 in both shapes must be equal, but are 400 and 100. Shapes are [1,400] and [1,100]. for '{{node tf.concat_17/concat}} = ConcatV2[N=3, T=DT_FLOAT, Tidx=DT_INT32](Placeholder, Placeholder_1, Placeholder_2, tf.concat_17/concat/axis)' with input shapes: [1,1600,85], [1,400,85], [1,100,85], [] and with computed input tensors: input[3] = <2>.

Could you help me to fix it? Thank you very much!

[PINTO0309]

Same issue. #114
Please do not post the same issue multiple times.

$ rm yolox_nano_320x320.xml
$ cp yolox_nano_320x320_tf.xml yolox_nano_320x320.xml

MODEL=yolox_nano
H=320
W=320
openvino2tensorflow \
--model_path openvino/FP32/${MODEL}_${H}x${W}.xml \
--output_saved_model \
--output_pb \
--output_no_quant_float32_tflite \
--output_weight_quant_tflite \
--output_float16_quant_tflite \
--output_integer_quant_tflite \
--string_formulas_for_normalization 'data / 255' \
--output_integer_quant_type 'uint8' \
--output_tfjs \
--output_coreml \
--weight_replacement_config weight_replacement_config_${MODEL}.json

[asizdzbest]

I am sorry if I bother you. I retrained the yolox-nano model with one class then convert new model to openvino, but I can not see yolox_nano_320x320_tf.xml or yolox_nano_320x320_vino.xml file when runing:
MODEL=yolox_nano H=320 W=320 $INTEL_OPENVINO_DIR/deployment_tools/model_optimizer/mo.py \ --input_model ${MODEL}_${H}x${W}.onnx \ --data_type FP32 \ --output_dir openvino/${MODEL}/${H}x${W}/FP32
What wrong with the script?

[PINTO0309]

I don't always know the details of what you do and what your problems are. Therefore, try to describe the work you have done in as much detail as possible. I am not a psychic.

This issue has been closed, please submit another issue.

[alexanderfrey]

Hi,

I'm running into the same issue. I trained YOLOX Nano with 2 classes and converted with the export_onnx.py script that is in the YOLOX repo. After that I use the openvino convert script as described in your blog post here:

https://qiita.com/PINTO/items/ed06e03eb5c007c2e102#6-8-onnx---openvino-ir-conversion

After that I end up with three files: experiment_opt.bin, experiment_opt.xml, experiment_opt.mapping

Converting the xml with:

openvino2tensorflow --model_path openvino/416x416/FP32/experiment_opt.xml --model_output_path saved_model_416x416 --string_formulas_for_normalization 'data / 255' --output_integer_quant_type 'uint8' --output_edgetpu --weight_replacement_config ../PINTO_model_zoo/132_YOLOX/weight_replacement_config_yolox_nano.json

Results in the following error:

`{'659': {'layer_id': '659', 'replace_mode': 'direct', 'values': [0, 1, 2]}}
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: Placeholder:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: Placeholder:0
ERROR: Dimension 1 in both shapes must be equal, but are 676 and 169. Shapes are [1,676] and [1,169]. for '{{node tf.concat_17/concat}} = ConcatV2[N=3, T=DT_FLOAT, Tidx=DT_INT32](Placeholder, Placeholder_1, Placeholder_2, tf.concat_17/concat/axis)' with input shapes: [1,2704,7], [1,676,7], [1,169,7], [] and with computed input tensors: input[3] = <2>.`

YOLOX_Nano has 2 classes. I use pytorch 1.9.0 and tensorflow 2.6.0rc2

You repeatedly mention to cp yolox_nano_320x320_tf.xml yolox_nano_320x320.xml.
My question is where do I get that yolox_nano_320x320_tf.xml file ?

Do need to get my model ? I can send it to you if it helps.

@PINTO0309

Thanks for any help.
Alexander

[alexanderfrey]

Apparently you are using <data axis="1"/> instead of <data axis="2"/> in layer 658. This is not documented anywhere, is it ?

Changing that I can compile the model. Now I'm getting the following WARNINGS when compiling with the --output_edgetpu flag:

`WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.nn.silu_47/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.nn.silu_54/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.nn.silu_47/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.nn.silu_54/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.nn.silu_47/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.nn.silu_54/IdentityN:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0`

Any ideas ? Currently I try to retrain with relu instead of silu activation function.

Best,Alexander

[PINTO0309]

@alexanderfrey

This is not documented anywhere, is it ?

Yes. However, I have been told that the OpenVINO xml file that is downloaded from here contains _tf.xml files. This repository does not have jurisdiction over independently created models that are not committed to this repository.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/132_YOLOX

Changing that I can compile the model. Now I'm getting the following WARNINGS when compiling with the --output_edgetpu flag

There is no problem ignoring it. This indicates that there is a problem with the conversion of ResizeBilinear / ResizeNearestNeighbor in the EdgeTPU compiler, so a workaround is being applied inside the tool. Conversely, an error will occur if you try to convert to the EdgeTPU model without this warning message being displayed.

It is correct that half_pixel_centers should be true, but the EdgeTPU compiler does not allow it. This is probably a bug in the compiler. Therefore, my conversion tool forces a workaround for the EdgeTPU compiler bug and forces the conversion to False. In other words, the warning text informs you that the inference performance of the model may be degraded due to minor differences from the correct model.

Currently I try to retrain with relu instead of silu activation function.

This is not a problem because it has nothing to do with the content of the warning message. However, please keep in mind that errors encountered in the conversion of individually customized models may require you to read the error messages carefully yourself and adjust some parts of the xml file.

In all of the issues related to this, you guys have not provided me with the .xml or .bin of your own generated models, so I am very confused as to exactly where the problem is occurring.

I am getting a little tired and will close this issue immediately if a similar issue is posted in the future.