Cannot convert YoloV5s to EdgeTPU #21
Comments
|
The error message can be ignored. The saved_model should be output. Use |
|
According to your suggestion, I ran the command: openvino2tensorflow --model_path best_opt.xml --model_output_path Models --output_saved_model True --output_h5 True --output_pb True --output_edgetpu TrueSo, without any --output_integer_quant_tflite. These were the generated files,
The output from the command was the following: .
.
.
__________________________________________________________________________________________________
tf.identity_2 (TFOpLambda) (1, 80, 3, 6, 80) 0 tf.compat.v1.transpose_2[0][0]
==================================================================================================
Total params: 7,233,672
Trainable params: 7,233,672
Non-trainable params: 0
__________________________________________________________________________________________________
TensorFlow/Keras model building process complete!
saved_model output started ==========================================================
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.math.multiply_49/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.math.multiply_56/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.math.multiply_49/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.math.multiply_56/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.math.multiply_49/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: tf.math.multiply_56/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: inputs:0
ERROR: can't pickle module objects
Traceback (most recent call last):
File "/home/luis/anaconda3/envs/yolo2lite/bin/openvino2tensorflow", line 1754, in convert
tf.saved_model.save(model, model_output_path)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1033, in save
obj, signatures, options, meta_graph_def)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1198, in _build_meta_graph
return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 1163, in _build_meta_graph_impl
asset_info.asset_index)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 755, in _serialize_object_graph
saveable_view.function_name_map)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py", line 800, in _write_object_proto
metadata=obj._tracking_metadata)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 3079, in _tracking_metadata
return self._trackable_saved_model_saver.tracking_metadata
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py", line 55, in tracking_metadata
return json_utils.Encoder().encode(self.python_properties)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 41, in python_properties
return self._python_properties_internal()
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/model_serialization.py", line 35, in _python_properties_internal
metadata = super(ModelSavedModelSaver, self)._python_properties_internal()
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 59, in _python_properties_internal
metadata.update(get_config(self.obj))
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/layer_serialization.py", line 118, in get_config
config = generic_utils.serialize_keras_object(obj)['config']
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/utils/generic_utils.py", line 245, in serialize_keras_object
config = instance.get_config()
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py", line 650, in get_config
return copy.deepcopy(get_network_config(self))
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 215, in _deepcopy_list
append(deepcopy(a, memo))
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 240, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 220, in <listcomp>
y = [deepcopy(a, memo) for a in x]
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 150, in deepcopy
y = copier(x, memo)
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 220, in _deepcopy_tuple
y = [deepcopy(a, memo) for a in x]
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 220, in <listcomp>
y = [deepcopy(a, memo) for a in x]
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/copy.py", line 169, in deepcopy
rv = reductor(4)
TypeError: can't pickle module objects
Switch to the output of an optimized protocol buffer file (.pb).
.pb output started ==================================================================
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.math.multiply_49/Mul:0
WARNING: The weights after Upsampling (tf.compat.v1.image.resize_nearest_neighbor) are shifted to the upper left. If you do not need to generate EdgeTPU models, set --output_edgetpu False and run again. OP: model/tf.math.multiply_56/Mul:0
.pb output complete! - Models/model_float32.pb
WARNING:tensorflow:From /home/luis/anaconda3/envs/yolo2lite/bin/openvino2tensorflow:1837: simple_save (from tensorflow.python.saved_model.simple_save) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.simple_save.
WARNING:tensorflow:From /home/luis/anaconda3/envs/yolo2lite/lib/python3.6/site-packages/tensorflow/python/saved_model/signature_def_utils_impl.py:201: build_tensor_info (from tensorflow.python.saved_model.utils_impl) is deprecated and will be removed in a future version.
Instructions for updating:
This function will only be available through the v1 compatibility library as tf.compat.v1.saved_model.utils.build_tensor_info or tf.compat.v1.saved_model.build_tensor_info.
Optimized graph converted to SavedModel! - Models
TFDS download started ===============================================================
TFDS download complete!
Full Integer Quantization started ===================================================
Full Integer Quantization complete! - Models/model_full_integer_quant.tflite
EdgeTPU convertion started ==========================================================
ERROR:
Traceback (most recent call last):
File "/home/luis/anaconda3/envs/yolo2lite/bin/openvino2tensorflow", line 2053, in convert
stderr=subprocess.PIPE).decode('utf-8')
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/home/luis/anaconda3/envs/yolo2lite/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['edgetpu_compiler', '-o', 'Models', '-s', 'Models/model_full_integer_quant.tflite']' returned non-zero exit status 1.
--------------------------------------------------------------------------------
Please install edgetpu_compiler according to the following website.
https://coral.ai/docs/edgetpu/compiler/#system-requirements
All the conversion process is finished! =============================================As you can see the TPU compilation failed even though and I have installed the edgetpu_compiler cmd tool and did not write any --output_integer_quant_tflite. I checked the generated "model_full_integer_quant.tflite" in Netron and again it seems to be a valid model. Any clues? Do you know if is possible to convert the yolov5s model using your strategy? One other question is ot possible to suppress the "TFDS download" phase? I had to install 25 Gigas of stuff and it took quite some time. I tried some argument to the openvino2tensorflow call but could not skip the download phase. |
|
I have committed the conversion script and the converted model here. However, to avoid the EdgeTPU compiler error, you need to manually eliminate the OP that causes the EdgeTPU compiler error. For example, the Post-Process part is very complex, so I often remove it. Also, the EdgeTPU compiler seems to abort models with large input resolutions.
I am considering making it possible to read Numpy format datasets as an additional feature in the future. This is not supported at the moment. If you don't want to download a large dataset, there is a way to generate your own Numpy file. The number of images required for calibration during quantization is about 100, so it is by far the lightest. For example, you can create your own script that modifies the following range to read image data from a Numpy(.npy) file and perform full INT8 quantization. def representative_dataset_gen():
for image in raw_test_data.take(10):
image = image['image'].numpy()
image = tf.image.resize(image, (height, width))
image = image[np.newaxis,:,:,:]
image = image - 127.5
image = image * 0.007843
yield [image]The following is a good reference on how to create a .npy file. Converts image data stored in a specific folder to a .npy file. import os
import sys
from glob import glob
from PIL import Image
import numpy as np
def resize(all_file):
tmp = None
for f in all_file:
img = Image.open(f)
nparray = np.array(img)
nparray = nparray[np.newaxis, :, :, :]
if tmp is not None:
tmp = np.vstack((tmp, nparray))
print("tmp.shape=", tmp.shape)
np.save('calibration_data_img', tmp)
else:
tmp = nparray.copy()
print("tmp.shape=", tmp.shape)
np.save('calibration_data_img', tmp)
path = "./images/"
all_file = os.listdir(path)
all_file = [os.path.join(path, f) for f in all_file]
resize(all_file)
print("Finish!!")Alternatively, the default dataset is MS-COCO, which is large, but you can choose a lighter dataset from the URL below. --ds_name_for_tfds_for_calibration DS_NAME_FOR_TFDS_FOR_CALIBRATION
Dataset name for TensorFlow Datasets for calibration.
https://www.tensorflow.org/datasets/catalog/overview |
|
Indeed it was the resolution. My model accepted an images of size (3640640). I repeated all steps using now a resolution of (3320320) and was able to compile it to the TPU. Command run: openvino2tensorflow --model_path best_opt.xml --model_output_path Models --output_saved_model True --output_h5 True --output_pb True --output_edgetpu TrueResult log: .
.
.
Full Integer Quantization started ===================================================
Full Integer Quantization complete! - Models/model_full_integer_quant.tflite
EdgeTPU convertion started ==========================================================
Edge TPU Compiler version 15.0.340273435
Model compiled successfully in 4144 ms.
Input model: Models/model_full_integer_quant.tflite
Input size: 7.33MiB
Output model: Models/model_full_integer_quant_edgetpu.tflite
Output size: 7.47MiB
On-chip memory used for caching model parameters: 6.74MiB
On-chip memory remaining for caching model parameters: 1.75KiB
Off-chip memory used for streaming uncached model parameters: 369.88KiB
Number of Edge TPU subgraphs: 1
Total number of operations: 264
Operation log: Models/model_full_integer_quant_edgetpu.log
Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 255
Number of operations that will run on CPU: 9
Operator Count Status
QUANTIZE 13 Mapped to Edge TPU
MUL 76 Mapped to Edge TPU
CONV_2D 70 Mapped to Edge TPU
RESIZE_NEAREST_NEIGHBOR 2 Mapped to Edge TPU
RESHAPE 3 Mapped to Edge TPU
PRELU 8 Mapped to Edge TPU
STRIDED_SLICE 6 Only Strided-Slice with unitary strides supported
CONCATENATION 14 Mapped to Edge TPU
TRANSPOSE 3 Operation not supported
ADD 66 Mapped to Edge TPU
MAX_POOL_2D 3 Mapped to Edge TPU
EdgeTPU convert complete! - Models/model_full_integer_quant_edgetpu.tflite
All the conversion process is finished! =============================================The resulting network after the TPU compilation is as follows: One question though. According to the command log, the operations "STRIDED_SLICE" and "TRANSPOSE" are run on the CPU. I have been told that, per inference stage, if the TPU caught an incompatible operation (CPU compatible) the rest of the operations would be run on the CPU even if compatible with the TPU. If that is true, my whole model will run on the CPU since the "STRIDED_SLICE" operation is the first one. Is this true? I find it very strange, but since I have no experience in this filed I just accepted it. Regarding the manual removal of OP that you mentioned, just for curiosity, how can you do it while keeping the model functional? Let me just thank you for help so far and the thorough guide for the Pythorch-Tensorflow conversion. amazing job! |
Presumably all OPs run on the CPU, but I haven't tried that either. For now, I am more interested in YoloV4 than YoloV5.
This is just an example.
|
|
I have tried the YoloV4-tiny-relu from here which is implemented in Tensorflow. After quantizing the model and convert it to TPU it gives 31 operation in CPU Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 101
Number of operations that will run on CPU: 31
Operator Count Status
PAD 2 Mapped to Edge TPU
SPLIT 7 Mapped to Edge TPU
ADD 6 Mapped to Edge TPU
SUB 6 Mapped to Edge TPU
QUANTIZE 27 Mapped to Edge TPU
QUANTIZE 6 Operation is otherwise supported, but not mapped due to some unspecified limitation
DEQUANTIZE 6 Operation is working on an unsupported data type
EXP 6 Operation is working on an unsupported data type
CONCATENATION 9 Mapped to Edge TPU
MAX_POOL_2D 3 Mapped to Edge TPU
CONV_2D 21 Mapped to Edge TPU
LOGISTIC 2 Mapped to Edge TPU
MUL 18 Mapped to Edge TPU
RESIZE_BILINEAR 1 Operation version not supported
SPLIT_V 12 Operation not supportedDo you know of any YoloV4-tiny implementation fully compatible to TPU? |
|
I have removed Post-Process, but the conversion is clean. |
|
If I want to train my custom data set on a yolov4-tiny and arrive to the same edgetpu model as you presented, what should I do? Is the following a possible solution?
Do you have a better alternative? |
|
Use the following repository to train, adjust the output layer, and output to For example, the following is easy to handle. Then you quantize |
|
Thank you, I will definitely give it a try. Meanwhile, I tried to edgetpu_compile your model yolov4_tiny_voc_416x416_full_integer_quant.tflite from here but got the error "Model not quantized". $ edgetpu_compiler -s yolov4_tiny_voc_416x416_full_integer_quant.tflite
Edge TPU Compiler version 15.0.340273435
Model compiled successfully in 161 ms.
Input model: yolov4_tiny_voc_416x416_full_integer_quant.tflite
Input size: 5.77MiB
Output model: yolov4_tiny_voc_416x416_full_integer_quant_edgetpu.tflite
Output size: 5.78MiB
On-chip memory used for caching model parameters: 3.00KiB
On-chip memory remaining for caching model parameters: 8.03MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 65
Operation log: yolov4_tiny_voc_416x416_full_integer_quant_edgetpu.log
Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 2
Number of operations that will run on CPU: 63
Operator Count Status
PAD 1 More than one subgraph is not supported
PAD 1 Mapped to Edge TPU
CONCATENATION 7 More than one subgraph is not supported
LEAKY_RELU 19 Operation not supported
SPLIT 3 More than one subgraph is not supported
MAX_POOL_2D 3 More than one subgraph is not supported
CONV_2D 1 Mapped to Edge TPU
CONV_2D 20 More than one subgraph is not supported
QUANTIZE 9 More than one subgraph is not supported
RESIZE_NEAREST_NEIGHBOR 1 Operation version not supported
Maybe I got it wrong, but I thought that by using your yolov4-tiny model I could get a fully TPU compilation like the one you presented. I am missing something? If it is relevant I used: |
|
Try quantization using $ sudo pip3 uninstall tensorboard-plugin-wit tb-nightly tensorboard \
tf-estimator-nightly tensorflow-gpu \
tensorflow tf-nightly tensorflow_estimator -y
$ sudo pip3 install tf-nightly |
|
Indeed tf-nightly solved most of the problems, however, RESIZE_NEAREST_NEIGHBOR and LEAKY_RELU could not be compiled. To achieve a full TPU compilation, from the .h5 and .json model here, I did:
The result is the following:
Now, I have to train the Yolov4-tiny implementation that you suggested and see if I can come up with the same compilation. As always thanks for the support @PINTO0309 |
|
This tool of mine has options available from the beginning to help you solve the problems you are having. It will automatically adjust The current latest version is v1.7.0, so it is recommended to update to the latest package.
|
|
Thanks @PINTO0309 guess I was reinventing the wheel xD I finally was able to train a Yolov4-tiny model from your recommended link (https://github.com/bubbliiiing/yolov4-tiny-tf2). I had to change some part of the code though, like removing some output and input layers that were unnecessary. Then I was able to convert the saved_model to openvino and then use your conversion tool to make the compilation to the TPU. From Netron the output is this:
I was comparing this model from the one you posted, and I noticed that my two right most output layers do not have the same dimentions as your model. I have not removed any Post-processing as far as I know. Could it be that?
One other thing, what is the meaning of each output layer, I will need to know that when running inference with this model right? I I am expecting the outputs to be the bounding boxes, confidence scores and class names but each one is each? I am new to this world of Deep Learning specially to deployment in TPUs so, sorry for the stupid question probably. |
|
I would recommend using YOLOv5-Lite. |
1. OS you are using e.g. Ubuntu 20.04, WIndows10, etc
Ubutnu 18.0.4
2. OS Architecture e.g. x86_64, armv7l, aarch64, etc
x86_64
3. Version of OpenVINO e.g. 2021.2.185, etc
2021.2.185
4. Version of TensorFlow e.g. v2.4.1, tf-nightly==2.5.0.dev20210128, etc
v2.3.1 and v.2.4.1 tried
5. Download URL for ONNX model, OpenVINO model, pt checkpoint, Tensorflow convertions
https://drive.google.com/drive/folders/1Etu0P_ioTFPCK6AmeCufSYADpJ-a6FjE?usp=sharing
13. Issue Details
I am trying to convert the pytorch model Yolov5s, implemented from ultralitics, to tflite to further compile it to a TPU.
I have trained this model on my dataset resulting in the checkpoint "best.pt". I converted this .pt file to the ONNX format using the ultralitics package, as specified here. After that I followed your guide, meaning I optimized the ONNX model, then converted the optimized ONNX to OpenVINO and finally to Tensorflow.
My first problem is in the last step, when using openvino2tensorflow. I cannot generate the SavedModel nor the h5 formats.
When I run the command:
The error part is the following:
As you can see it fails to convert to a SavedModel with the error "TypeError: can't pickle module objects". Do you have any idea what might be the problem?
The .pb files are generated and the quantization version are generated as well. I even tried to compile the "output_integer_quant_tflite" and "output_full_integer_quant_tflite" tflite version using the edgetpu_compiler however it Aborts the compilation.
$ edgetpu_compiler -sa Models/model_integer_quant.tflite Edge TPU Compiler version 15.0.340273435 Internal compiler error. Aborting!I have also tried to open the resulting .xml file, from the openvino conversion, using the application Netron, and the Network seems to be normal. Here are some pictures of the begging and end of my xml.
Beginning
Ending
Do you have any idea what might be causing the SavedModel conversion error and then the EdgeTPU compilation failure? I suspect the second is related to the first.
Any sugestion is more than welcome!
The text was updated successfully, but these errors were encountered: