Reland Reland "Port DW Pose preprocessor"

[huchenlei]

Relands #1890.

This PR now makes onnx runtime optional. (Not forcing user to install it on startup)

[lllyasviel]

where is the onnx dependency? does webui has onnx in it? or pytorch has onnx in it?

[lllyasviel]

ah nvm. just find out check_onnx_environment():
but can we make it simpler? does opencv has onnx inference in it? I saw 2 years ago cv2 has some neural network things seems to use onnx

[huchenlei]

WebUI only has 1 onnx ref on a non-essential feature:
https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/68f336bd994bed5442ad95bad6b6ad5564a5409a/modules/textual_inversion/autocrop.py#L299

The onnx environment check code comes from diffusers implementation:
https://github.com/huggingface/diffusers/blob/f0725c5845b150b3042e9e98b1cda154d3bef5b5/src/diffusers/utils/import_utils.py#L151

I don't think opencv depends on onnx right now. Searching onnx in opencv repo results nothing.

[lllyasviel]

but some people seem to claim that opencv dnn (cv2.dnn) can read and infer onnx on cpu

[SLAPaper]

FYI, cv2.dnn API that seems to read ONNX model: https://docs.opencv.org/4.8.0/d6/d0f/group__dnn.html#ga7faea56041d10c71dbbd6746ca854197

[yzd-v]

We support to run onnx model with opencv in branch opencv_onnx. But we need to manually build OpenCV through cmake to work with GPU. However, according to our test, the speed is also fine with CPU.

[SLAPaper]

got error when using opencv 4.5.5:

[ERROR:[email protected]] global D:\a\opencv-python\opencv-python\opencv\modules\dnn\src\onnx\onnx_importer.cpp (909) cv::dnn::dnn4_v20211220::ONNXImporter::handleNode DNN/ONNX: ERROR during processing node with 1 inputs and 1 outputs: [Identity]:(onnx::Clip_1106) from domain='ai.onnx'
Traceback (most recent call last):
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "C:\Users\SLAPaper\scoop\apps\miniconda3\current\envs\sd_webui\lib\site-packages\anyio\to_thread.py", line 28, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(func, *args, cancellable=cancellable,
  File "C:\Users\SLAPaper\scoop\apps\miniconda3\current\envs\sd_webui\lib\site-packages\anyio\_backends\_asyncio.py", line 818, in run_sync_in_worker_thread
    return await future
  File "C:\Users\SLAPaper\scoop\apps\miniconda3\current\envs\sd_webui\lib\site-packages\anyio\_backends\_asyncio.py", line 754, in run
    result = context.run(func, *args)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\controlnet_ui\controlnet_ui_group.py", line 609, in run_annotator
    result, is_image = preprocessor(
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 79, in decorated_func
    return cached_func(*args, **kwargs)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\utils.py", line 67, in cached_func
    return func(*args, **kwargs)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\global_state.py", line 35, in unified_preprocessor
    return preprocessor_modules[preprocessor_name](*args, **kwargs)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\processor.py", line 261, in run_model
    return remove_pad(self.model_openpose(
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\__init__.py", line 358, in __call__
    poses = self.detect_poses_dw(oriImg)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\__init__.py", line 331, in detect_poses_dw
    self.load_dw_model()
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\__init__.py", line 215, in load_dw_model
    self.dw_pose_estimation = Wholebody(onnx_det, onnx_pose)
  File "C:\Users\SLAPaper\workspace\stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\openpose\wholebody.py", line 24, in __init__
    self.session_pose = cv2.dnn.readNetFromONNX(onnx_pose)
cv2.error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\dnn\src\onnx\onnx_importer.cpp:928: error: (-2:Unspecified error) in function 'cv::dnn::dnn4_v20211220::ONNXImporter::handleNode'
> Node [[email protected]]:(onnx::Clip_1106) parse error: OpenCV(4.5.5) D:\a\opencv-python\opencv-python\opencv\modules\dnn\src\dnn.cpp:6017: error: (-215:Assertion failed) inputs.size() in function 'cv::dnn::dnn4_v20211220::Layer::getMemoryShapes'
>

upgrade to 4.8.0 then everything went fine