[1.1.427] New preprocessor depth_hand_refiner

[huchenlei]

Thanks Fannovel16 for his hard work extracting dependencies out for hand refiner in https://github.com/Fannovel16/comfyui_controlnet_aux/.

Previously users need to use extensions like https://github.com/jexom/sd-webui-depth-lib to pick a matching hand gesture and move it to a correct location. depth_hand_refiner preprocess now does this job automatically for you.

How to use

Now you can manually draw the inpaint mask on hands and use a depth ControlNet unit to fix hands with following steps:

Step 1: Generate an image with bad hand.

Step 2: Switch to img2img inpaint. Draw inpaint mask on hands.

Step 3: Enable ControlNet unit and select depth_hand_refiner preprocessor.

Step 4: Generate

ADetailer usage example (Bing-su/adetailer#460):

You need to wait for ADetailer author to merge that PR or checkout the PR manually. This section is independent of previous img2img inpaint example. Here the generation is txt2img.

Image generated without ADetailer

Image generated with ADetailer

ADetailer setting

Make sure you adjust denoising strength so that depth map can take control of hand rendering.

Known issues

Hand refiner cannot handle complex hand gestures such as crossed fingers. Example: only 1 hand detected when finger are crossed:

[wardensc2]

My SD inpaint tab lost function after the latest update please fix it. Thank you so much

[wardensc2]

My SD inpaint tab lost function after the latest update please fix it. Thank you so much

I already fixed. Not your update fault, it's A1111 latest update

[Woisek]

I did an update right now, but depth_hand_refiner is not in the list ... ? 🤔
Anything else we need to do to get it?

[Sisimshow]

It seems that if the Hand Refiner specific depth model is selected in ControlNet model on Adetailer, you're unable to select hand_depth_refiner under ControlNet module. I believe this should be added to improve quality.
https://huggingface.co/hr16/ControlNet-HandRefiner-pruned

[Woisek]

Thanks, I will try that. 👍

[Woisek]

Thanks again, this is really helpful! After some trying, it can really correct hands to a point, where it either fits or can be easily photoshopped. A great addition.

[zcatharisis]

I completely misunderstood your point. You're right, adetailer treats the model you linked as an inpainting model, not a depth model, which means you don't get access to the hand refiner preprocessor. I thought you were talking about not being able to select the preprocessor in the controlnet extension's model, not the adetailer controlnet module.

Honestly I should have been blasted for this misinfo, my mistake.

Quick workaround to fix this: rename "control_sd15_inpaint_depth_hand_fp16.safetensors" to "control_sd15_depth_hand_fp16.safetensors"

[Sisimshow]

Glad I wasn't crazy lol

edit

Oh, and thanks for the fix :) simple enough

[zcatharisis]

It seems that if the Hand Refiner specific depth model is selected in ControlNet model on Adetailer, you're unable to select hand_depth_refiner under ControlNet module. I believe this should be added to improve quality. https://huggingface.co/hr16/ControlNet-HandRefiner-pruned

But why though. You're already using adetailer to inpaint the hands, why force another pass in the controlnet module. If you wanted to do that anyway, just run the output through img2img at 0 denoising.

[Sisimshow]

Perhaps I just misunderstand, but doesn't this use ControlNet through Adetailer? In the screenshots above it says to select the ControlNet depth model and the hand refiner module. The model I posted is a depth model that specializes in hands, so I proposed being able to select it as a ControlNet model in Adetailer and still access the hand refiner module, as currently, it doesn't seem to allow that. I could be missing some details here, so forgive me if that's the case.

[zcatharisis]

Ah, what you're missing is that adetailer implements its own controlnet module when using the tab there, completely independent of the controlnet extension by mikubill. So while you may need to use the controlnet extension when using the inpainting feature, you don't even need to install it to use the adetailer's controlnet module.

[Sisimshow]

Ah got it, missed that bit, thank you :)

[wardensc2]

Mine is not work, controlnet keep asking about upscaler with inpaint image2image and all inpaint for hand is still bug

[rmcc3]

Fails to process correctly for me. Also tried deleting the venv.

*** Error running process: E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\controlnet.py
Traceback (most recent call last):
File "E:\stable-diffusion-webui-master\modules\scripts.py", line 771, in process
script.process(p, *script_args)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1092, in process
self.controlnet_hack(p)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\controlnet.py", line 1077, in controlnet_hack
self.controlnet_main_entry(p)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\controlnet.py", line 908, in controlnet_main_entry
detected_map, is_image = preprocessor(
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\utils.py", line 75, in decorated_func
return cached_func(*args, **kwargs)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\utils.py", line 63, in cached_func
return func(*args, **kwargs)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\global_state.py", line 37, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\processor.py", line 685, in run_model
self.load_model()
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\scripts\processor.py", line 671, in load_model
self.model = MeshGraphormerDetector.from_pretrained(
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\annotator\hand_refiner_portable\hand_refiner_init_.py", line 17, in from_pretrained
pipeline = MeshGraphormerMediapipe(args)
File "E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\annotator\hand_refiner_portable\hand_refiner\pipeline.py", line 198, in init
self.detector = vision.HandLandmarker.create_from_options(options)
File "E:\stable-diffusion-webui-master\venv\lib\site-packages\mediapipe\tasks\python\vision\hand_landmarker.py", line 364, in create_from_options
return cls(
File "E:\stable-diffusion-webui-master\venv\lib\site-packages\mediapipe\tasks\python\vision\core\base_vision_task_api.py", line 70, in init
self._runner = _TaskRunner.create(graph_config, packet_callback)
RuntimeError: Unable to open file at E:\stable-diffusion-webui-master\venv\lib\site-packages/E:\stable-diffusion-webui-master\extensions\sd-webui-controlnet\annotator\hand_refiner_portable\hand_refiner\hand_landmarker.task, errno=22

[huchenlei]

This is really wield. It seems like 2 absolute paths are concat together, but I cannot find the code that does concat.

[rmcc3]

Found the issue. This error only happens when hrnetv2_w64_imagenet_pretrained.pth or graphormer_hand_state_dict.bin is missing from sd-webui-controlnet\annotator\hand_refiner_portable\hand_refiner.

Weird that it does that concat. Looks like it is related to custom_hf_download in util.py in hand_refiner_portable\hand_refiner. Better error handling might make this less confusing.

[wardensc2]

I still dont know how to use it even using latest update of adetail still cant get any correct hand properly ><

[dayunbao]

How did you use such a high denoising strength, without supplying a separate prompt for the hands, and end up with a hand? I end up with a copy of the main image in place of the hand. The high denoising strength tries to reproduce the whole prompt into the space where the hands used to be.

Right now, this is destroying hands, not making them better. Am I doing something wrong>

[huchenlei]

Do you mean generation result like that?

[dayunbao]

Yes

[dayunbao]

This also happens:
Without ADetailer

With ADetailer

[huchenlei]

Well, I find it work better in realistic/2.5D models. I think your last example's issue is caused by handrefiner either not detecting hand or generated wrong hand depth map.

[dayunbao]

OK. I think I know what's going on. The dw_openpose_full preprocessor is better at detecting hands than the depth_hand_refiner. I tried running the depth_hand_refiner on the same image I gave to dw_openpose_full, and it failed. It might be better to use the two in combination somehow, where the bounding boxes for the hands is based on the hand keypoints found by dw_openpose_full. But I'm not familiar with how depth_hand_refiner is detecting hands.

[DavideAlidosi]

After update to 427 I have this error on startup:

*** Error running install.py for extension L:\WebUI\webui\extensions\sd-webui-controlnet.
*** Command: "L:\WebUI\system\python\python.exe" "L:\WebUI\webui\extensions\sd-webui-controlnet\install.py"
*** Error code: 1
*** stderr: L:\WebUI\webui\extensions\sd-webui-controlnet\install.py:3: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
***   import pkg_resources
*** git: 'submodule' is not a git command. See 'git --help'.
*** Traceback (most recent call last):
***   File "L:\WebUI\webui\extensions\sd-webui-controlnet\install.py", line 66, in <module>
***     sync_submodules()
***   File "L:\WebUI\webui\extensions\sd-webui-controlnet\install.py", line 16, in sync_submodules
***     subprocess.run(["git", "submodule", "init"], check=True, cwd=repo_root)
***   File "subprocess.py", line 524, in run
*** subprocess.CalledProcessError: Command '['git', 'submodule', 'init']' returned non-zero exit status 1.

[huchenlei]

#2406 should unblock you.

[DavideAlidosi]

That's make the job, thanks a lot!
Of couse now on the startup I have this new warning:

Warning: ControlNet failed to sync submodules. Please try run git submodule init and git submodule update manually.

[GrantWasil]

I am also having the same warning shown when I launch the application:

Installing requirements
Command '['git', 'submodule', 'init']' returned non-zero exit status 1.
Warning: ControlNet failed to sync submodules. Please try run `git submodule init` and `git submodule update` manually.

[huchenlei]

I will take further look into this. Probably need to ref how a1111 handles submodules. One potential cause is PATH environment variables. See:

https://stackoverflow.com/questions/54672008/submodule-appears-to-be-a-git-command-but-we-were-not-able-to-execute-it

[wiseoldowl-66]

I'm also experiencing this error, although the commands complete if run manually.

[wardensc2]

[huchenlei]

I essentially just copy your img2img inpaint settings.

[wardensc2]

Thank I just find out how to use this. So basically this is no magical fix, we still need to re-render a few times to get a better result

[huchenlei]

Yes, this is not magical. Previously people uses depth lib to pick hand depth map they want to use to fix hands. HandRefiner just automates that process.

[wardensc2]

So adetailer not helping that much due to we still have to choose correct denoise strength and step and see the result a few times and inpaint is much faster than adetailer due to it need to render the whole image first

[wenquanlu]

I think using finetuned weights for inpainting https://huggingface.co/hr16/ControlNet-HandRefiner-pruned/blob/main/control_sd15_inpaint_depth_hand_fp16.safetensors can make the generated images more closely follow the depth map

[Woisek]

Be aware that for this to work, the play between denoise and steps is important and this is mostly not a "one round" fix.
Low denoise -> higher settings, high denoise -> lower steps.
Depending on the grade of misform, start with a higher denoise (0.5 to 0.7) and low steps (20 to 30 max.), with each continuing correction lowering the denoise and up the steps.

EDIT: And yes, do only one hand at a time, reimporting the best result and continue.

Left original, right corrected:

[E2GO]

So just to be sure - it's only for SD1.5 version and SDXL is not supported yet?

[huchenlei]

This is not a new model. This is a new preprocessor to generate depth map. For SDXL, you just need to use corresponding SDXL depth ControlNet model.

[E2GO]

Got it! Thanks for answer

[banjaminicc]

But to use the hand refining functioning in SDXL we would need a model retrained from the hand refiner right, since it's SD1.5 only

[LoRAMilkshake]

Inpaint area "whole picture" or "only masked"? which sampler?

[Adzido]

[stormcenter]

yes ,I meet same problem

[aimarxjg]

[huchenlei]

能贴一下你文件夹的路径吗？代码里的检测是检测到有文件就不会去下载的

def custom_hf_download(pretrained_model_or_path, filename, cache_dir, subfolder='', use_symlinks=False):
    local_dir = os.path.join(cache_dir, pretrained_model_or_path)
    model_path = os.path.join(local_dir, *subfolder.split('/'), filename)
    
    if not os.path.exists(model_path):
        print(f"Failed to find {model_path}.\n Downloading from huggingface.co")
    ...

[huchenlei]

应该没问题啊。你实在不行把这个路径输到python shell里测一下？

import os
os.path.exists("F:\sd-webui-aki-V4.5\extensions\sd-webui-controlnet\annotator\downloads\hand_refiner\hr16/ControlNet-HandRefiner-pruned\graphormer_hand_state_dict.bin")

[Adzido]

If I use this after the hires fix should I use pixel perfect in CN? Or always without it and 512 res?
Can it work good with SDXL?

[wardensc2]

After testing for a while, I found down that the detection hand of this new preprocessor is pretty weak compare with other model, it has the ability to fix the deformed hand however it cant recognize hand in the image, it can only detect the hand in the image which have very good contrast with hand and the background or bright image (such as in a morning...) otherwise it can't find it. If someone cant combine the detection model of adetailer for example with this new preprocessor it will be must better.

So it's not like messing around with denoise and step to find the suitable hand, first it need to detect the correct region in the image is hand and create the depth map first.

[Woisek]

first it need to detect the correct region in the image is hand and create the depth map first.

That's ... why you mask the hand ...

[wardensc2]

Even I mark sometimes the preprocessor still not detecting the hand T_T

[ema7569]

The other possibility (seem more correct) is to download the three models and place them manually in webui folders
https://huggingface.co/hr16/ControlNet-HandRefiner-pruned/tree/main and place them in the 3 following folders:

1/ stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\hand_refiner\hr16\ControlNet-HandRefiner-pruned\graphormer_hand_state_dict.bin
2/ stable-diffusion-webui\extensions\sd-webui-controlnet\annotator\downloads\hand_refiner\hr16\ControlNet-HandRefiner-pruned\hrnetv2_w64_imagenet_pretrained.pth
3/ stable-diffusion-webui\extensions\sd-webui-controlnet\models\control_sd15_inpaint_depth_hand_fp16.safetensors

Restart the console and the webui

Now I can use the controlnet preview and see the depthmap:
In controlnet model select control_sd15_inpaint_depth_hand_fp16 and preprocessor depth_hand_refiner. Check upload independent image and allow preview. Add your image in controlnet image. Click run preprocessor. The depthmap will show.

The original

The depthmap

A simple layer test for correspondence

From here we can use the original controlnet depthmap or not. The problem is to find the corrects parameters :)
Note I tested with just one image.

Thank

Update: In another test it miss one hand detection

[wardensc2]

I try so many parameters, while SD can't just take the depth map and render it like the depth map library we used instead of so many weird images

[wardensc2]

Another thing is this preprocessor is good when detect image size 512x512 or 512x768, any kind out of these sizes become undetectable such as 1920x1080, 512x1024 and so on

[aimarxjg]

It only takes 2 files, sometimes it's normal to not detect the hand, and then it comes out after a few more tests. MHM. "control_sd15_inpaint_depth_hand_fp16.safetensors" This file is used by COMFYUI workflow.。It is not good for cartoon animation, more suitable for SD1.5 real person repair hands, so only 512 or 768. And SDXL repair hands rely on the slider LORA, that effect is good

[zcatharisis]

So really fun fact: you can change the model this uses for hand detection. Currently, it uses mediapipe, which as I understand is shit for recognizing gloved hands. So you can change the model from controlnet's very own openpose_hand preprocessor or even adetailer's hands.pt. hands.pt almost instantly provides the depthmap too, being only 20MB large; I'm guessing that the current 500MB model takes a while to load.

To replace with controlnet's openpose_hand

1. Go to sd.webui\webui\extensions\sd-webui-controlnet\annotator\downloads\openpose and copy hand_pose_model.pth to sd.webui\webui\extensions\sd-webui-controlnet\annotator\downloads\hand_refiner\hr16\ControlNet-HandRefiner-pruned
   NOTE: if the file doesn't exist, load up a1111, drag an image to controlnet, select openpose hand, then preview to download the file.
2. move hrnetv2_w64_imagenet_pretrained.pth to a different directory (or delete it).
3. Rename hand_pose_model.pth to hrnetv2_w64_imagenet_pretrained.pth

To replace with adetailer's hand.pt

1. Go to sd.webui\webui\models\adetailer and copy hand_yolov8s.pt to sd.webui\webui\extensions\sd-webui-controlnet\annotator\downloads\hand_refiner\hr16\ControlNet-HandRefiner-pruned
2. 2. move hrnetv2_w64_imagenet_pretrained.pth to a different directory (or delete it).
3. Rename hand_yolov8s.pt to hrnetv2_w64_imagenet_pretrained.pth
   NOTE: yes you are renaming the pt file to a pth one. This doesn't affect anything apparently.

I haven't done testing on which is more reliable. Maybe someone who can automate that can provide feedback on this.

[zcatharisis]

Why this works:
As per https://huggingface.co/hr16/ControlNet-HandRefiner-pruned/discussions/3, you don't actually need to make an openpose stick figure to get the depthmap. All you need is some reliable way of detecting hands, be it through mediapipe, openpose or boundingbox (yolov8, adetailer's hand model). Maybe an interface can let users select each one they prefer, but I'm no coder, so this hack is the best I can come up with.

[huchenlei]

Yes I do notice that the preprocessor loading/inference time is longer than average. Use better hand detection algorithm is very promising on speed improvement. However, we have limited bandwidth/Dev resource, so currently we are prioritizing implement more features first. Please submit an issue so that the task get tracked. Thanks!

[wardensc2]

I try both your methods: openpose and adetailer.

For openpose it running but the hand is destroy by many artifacts no matter how many times you change the denoise strength

For adetailer I get error message and hand is destroyed too same as openpose

So we need to figure the different ways

This is the original image:

This is hand with 0.5 strength denoise - openpose method:

Reverse back to original file same 0.5 denoise:

[zcatharisis]

@huchenlei. Will make an issue now, but yeah I don't expect it to be dealt with in the near future. After some testing, it seems like the reliability of detecting hands is the same across the three models (mediapipe, openpose, yolov8), sticking with the current mediapipe model should be fine. Anyone who comes across this thread is free to switch to yolov8 hand detection for speed and to save on memory.
The issue now lies in gloved hands (or perhaps more accurately, hands that are not sufficiently contrasted with the surrounding colors); it seems that even if the detector identifies gloved hands accurately, something goes wrong in the later processes that prevents a depthmap from being made. So put "gloves" in your negative prompts for now ig.

@wardensc2
Honestly I'm not gonna bother helping you here. Changing the hand detection model isn't going to change whatever's janky with your setup because it doesn't affect the inpainting process in the first place.

[huchenlei]

@wardensc2 The exception you see is because A1111's safety mechanism. It will block anything not on whitelist. You need to wrap the call under a context like following code to bypass the safety restriction.

sd-webui-controlnet/scripts/processor.py

Line 670 in a5b3fa9

with Extra(self.torch_handler):

Hopefully this unblocks you with your experiment.

[wardensc2]

@wardensc2 The exception you see is because A1111's safety mechanism. It will block anything not on whitelist. You need to wrap the call under a context like following code to bypass the safety restriction.

sd-webui-controlnet/scripts/processor.py

Line 670 in a5b3fa9

with Extra(self.torch_handler):

Hopefully this unblocks you with your experiment.

Thank I will try

PS: after trying to replace the code as you said, but the processor.py line 670 already had it. So Im not sure what can I do to wrap the call

[wardensc2]

@huchenlei. Will make an issue now, but yeah I don't expect it to be dealt with in the near future. After some testing, it seems like the reliability of detecting hands is the same across the three models (mediapipe, openpose, yolov8), sticking with the current mediapipe model should be fine. Anyone who comes across this thread is free to switch to yolov8 hand detection for speed and to save on memory. The issue now lies in gloved hands (or perhaps more accurately, hands that are not sufficiently contrasted with the surrounding colors); it seems that even if the detector identifies gloved hands accurately, something goes wrong in the later processes that prevents a depthmap from being made. So put "gloves" in your negative prompts for now ig.

@wardensc2 Honestly I'm not gonna bother helping you here. Changing the hand detection model isn't going to change whatever's janky with your setup because it doesn't affect the inpainting process in the first place.

@zcatharisis, I just give you some result about changing detection model, it still be the same images, just try to contribute some tests

[zcatharisis]

Sorry, but that kind of feedback on reliability isn't the helpful kind.

[wardensc2]

Just stop, I don't want to argue with you. Those feedbacks don't just mean for you, you are no coders your solutions aren't worked, accept it. If you don't find solutions let other people do.

[zcatharisis]

Like I said: changing the hand detection model will not fix how Graphormer reads the image, how Trimesh traces the MANO mesh it outputs, how the controlnet depth model you selected affixes itself to the unet of the checkpoint you use to create your ai waifu. I'm not a coder, but I can read.

And I did provide a solution to a problem, just one that doesn't fix your jank setup. At this point, just reinstall A1111, use default options, and try the extensions again.

[wardensc2]

Like I said: changing the hand detection model will not fix how Graphormer reads the image, how Trimesh traces the MANO mesh it outputs, how the controlnet depth model you selected affixes itself to the unet of the checkpoint you use to create your ai waifu. I'm not a coder, but I can read.

And I did provide a solution to a problem, just one that doesn't fix your jank setup. At this point, just reinstall A1111, use default options, and try the extensions again.

No your solution not working for anything, I already reinstall A1111 and test.

And you tell me that changing model not affect detection, not fix the output. So what is the point of changing ??? For Vram saving and speed, I dont feel anything change at all and I used 3090 by the way. Maybe it helping someone using lower vram card (no one show the result yet) but for me nothing change at all.

And I already install comfy and test. Using workflow of this guy:

https://www.youtube.com/watch?v=Tt-Fyn1RA6c&t=273s

When I change the prompt to something new, result is worse than in the video. This preprocessor maybe can help a little bit but the effect is not as much as people thought.

[zcatharisis]

So what is the point of changing ??? For Vram saving and speed, I dont feel anything change at all and I used 3090 by the way. Maybe it helping someone using lower vram card (no one show the result yet) but for me nothing change at all.

Congratulations, you read and quoted my post and managed to skip over where I explicitly said that. If you're still having trouble with this feature, I suppose this is a PEBCAK issue. Those are really hard to fix, good luck.

[zongmi]

depth_hand_refiner Most of the working hands become like this, please help me
PS: sdxl, adetailer, img2img

[zongmi]

ADetailer steps: 35,
ADetailer sampler: DPM++ 3M SDE Karras,

[wardensc2]

Dont use Adetailer just inpaint manually

[zongmi]

Can't it be used in adetailer?

[zongmi]

I tried it, and it will indeThank you, I tried it, and it will indeed not happen again, as long as it is not used in adetailered not happen again, as long as it is not used in adetailer

[zcatharisis]

Give adetailer another shot, but this time reduce controlnet weight to maybe 0.6. I had a similar issue with the hands getting too glassy when using adetailer, and this fixed it for me.

[MatthiasStockert]

I can’t get it to work on my Mac (M1 Max chip):

NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty.memory_format' is only available for these backends: [CPU, MPS, Meta, QuantizedCPU, QuantizedMeta, MkldnnCPU, SparseCPU, SparseMeta, SparseCsrCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

[misato2099]

Same problem here

[carnivar10]

Checkpoint : SDXL
Preprocessor : depth_hand_refiner
Model : controlnetxlCNXL_bdsqlszDepth, controlnetxlCNXL_diffDepthFull, kohya_controllllite_xl_depth, kohya_controllllite_xl_depth_anime

For 1.5 Checkpoint it works fine, but for SDXL it doesn't seem to be able to detect hands with depth, I've tested the various models I entered above and they all show the same thing, very little detection. Is there a similar case out there?

[carnivar10]

In the case of 1.5, it seems to work fine

[zongmi]

Check Pixel Perfect or set preprocessor resolution Try

[carnivar10]

Thanks I tried changing the resolution and it works fine

[suonnon]

Hi, where is the link to the download the hand refiner model?

[misato2099]

I am using MBP M1 Pro 16GB. There was error in running the preprocessor:

/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/scipy/sparse/_index.py:102: SparseEfficiencyWarning: Changing the sparsity structure of a csr_matrix is expensive. lil_matrix is more efficient.
self._set_intXint(row, col, x.flat[0])
Traceback (most recent call last):
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
output = await app.get_blocks().process_api(
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1431, in process_api
result = await self.call_function(
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1103, in call_function
prediction = await anyio.to_thread.run_sync(
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/gradio/utils.py", line 707, in wrapper
response = f(*args, **kwargs)
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/controlnet_ui/controlnet_ui_group.py", line 1013, in run_annotator
result, is_image = preprocessor(
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/utils.py", line 80, in decorated_func
return cached_func(*args, **kwargs)
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/utils.py", line 64, in cached_func
return func(*args, **kwargs)
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/global_state.py", line 37, in unified_preprocessor
return preprocessor_modules[preprocessor_name](*args, **kwargs)
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/processor.py", line 859, in run_model
self.load_model()
File "/Users/user/pinokio/api/automatic1111.git/app/extensions/sd-webui-controlnet/scripts/processor.py", line 845, in load_model
self.model = MeshGraphormerDetector.from_pretrained(
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/hand_refiner/init.py", line 17, in from_pretrained
pipeline = MeshGraphormerMediapipe(args)
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/hand_refiner/pipeline.py", line 76, in init
mesh_sampler = Mesh(device=args.device)
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/mesh_graphormer/modeling/_mano.py", line 124, in init
self._U = [u.to(device) for u in self._U]
File "/Users/user/pinokio/api/automatic1111.git/app/venv/lib/python3.10/site-packages/mesh_graphormer/modeling/_mano.py", line 124, in
self._U = [u.to(device) for u in self._U]
NotImplementedError: Could not run 'aten::empty.memory_format' with arguments from the 'SparseMPS' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty.memory_format' is only available for these backends: [CPU, MPS, Meta, QuantizedCPU, QuantizedMeta, MkldnnCPU, SparseCPU, SparseMeta, SparseCsrCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMeta, AutogradMTIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradNestedTensor, Tracer, AutocastCPU, AutocastCUDA, FuncTorchBatched, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PythonDispatcher].

CPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterCPU.cpp:31034 [kernel]
MPS: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterMPS.cpp:22748 [kernel]
Meta: registered at /dev/null:241 [kernel]
QuantizedCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterQuantizedCPU.cpp:929 [kernel]
QuantizedMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterQuantizedMeta.cpp:105 [kernel]
MkldnnCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterMkldnnCPU.cpp:507 [kernel]
SparseCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseCPU.cpp:1379 [kernel]
SparseMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseMeta.cpp:249 [kernel]
SparseCsrCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterSparseCsrCPU.cpp:1128 [kernel]
BackendSelect: registered at /Users/runner/work/pytorch/pytorch/pytorch/build/aten/src/ATen/RegisterBackendSelect.cpp:726 [kernel]
Python: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:144 [backend fallback]
FuncTorchDynamicLayerBackMode: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:491 [backend fallback]
Functionalize: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/FunctionalizeFallbackKernel.cpp:280 [backend fallback]
Named: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
Conjugate: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/ConjugateFallback.cpp:21 [kernel]
Negative: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/NegateFallback.cpp:23 [kernel]
ZeroTensor: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/ZeroTensorFallback.cpp:90 [kernel]
ADInplaceOrView: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/VariableFallbackKernel.cpp:63 [backend fallback]
AutogradOther: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradCPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradCUDA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradHIP: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradXLA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMPS: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradIPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradXPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradHPU: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradVE: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradLazy: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMeta: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradMTIA: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse1: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse2: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradPrivateUse3: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
AutogradNestedTensor: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/VariableType_2.cpp:17484 [autograd kernel]
Tracer: registered at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/generated/TraceType_2.cpp:16726 [kernel]
AutocastCPU: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:487 [backend fallback]
AutocastCUDA: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/autocast_mode.cpp:354 [backend fallback]
FuncTorchBatched: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:815 [backend fallback]
FuncTorchVmapMode: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/VmapModeRegistrations.cpp:28 [backend fallback]
Batched: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/LegacyBatchingRegistrations.cpp:1073 [backend fallback]
VmapMode: fallthrough registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
FuncTorchGradWrapper: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/TensorWrapper.cpp:210 [backend fallback]
PythonTLSSnapshot: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:152 [backend fallback]
FuncTorchDynamicLayerFrontMode: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:487 [backend fallback]
PythonDispatcher: registered at /Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:148 [backend fallback]

How to fix it? Thank you.

[BenDes21]

Hi, is it compatible with SDXL ? Thanks

[carnivar10]

It works fine, if you look at what I wrote at the above it might help you

[BenDes21]

sorry, forget to said that I would like to use hand refiner with Adetailer :/

Also, which models for Depth do you recommend for SDXL + hand refiner : https://huggingface.co/lllyasviel/sd_control_collection/tree/main

Thanks

[AlichinoML]

Hello,
Is there a way to change detection model confidence value (in code) for testing? I've tried to use hand_yolo9c.pt model instead of hrnetv2_w64_imagenet_pretrained.pth and it works but detection is worse compared to Adetailer with the same model and confidence value of 0.3.

[sp1cae]

使用flux 的 controlnet depth 与 depth_hand_refiner时 搭配adetailer不生效