[ControlNet 1.1] The updating track.

[lllyasviel]

We will use this repo to track some discussions for updating to ControlNet 1.1.

[lllyasviel]

Update: ControlNet 1.1 is released here.

[lllyasviel]

I think we can ignore cnet11 Tile model right now. We are not very sure how to make use of it.
The inpainting model may need more considerations in implementation and perhaps we just get other models first.

[continue-revolution]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning to connect my extension to your inpainting model.

So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

Edit on 2023/04/18 already connected. Checkout my extension readme for how to use.

[pkuliyi2015]

I think we can ignore cnet11 Tile model right now. We are not very sure how to make use of it. The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I have been long working on tiles. Have you tried cooperating with noise inversion tricks? I think this can be very good, with a better trained model it may be comparable to the quality of GigaGAN.

My extension is here -> https://github.com/pkuliyi2015/multidiffusion-upscaler-for-automatic1111

I will adapt your tile model to see the result and update it here.

[2blackbar]

Yes the tile model can be a saviour for upscaling and no doubles

[halr9000]

This thread is already amazing. ^ 3 amazing devs collaborating

[lllyasviel]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning connect my extension to your inpainting model.

So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

my gradio demo does not have masked diffusion in it. what is displayed now is just original results from standard non-masked diffusion. but masked diffusion will be better.

[pkuliyi2015]

The model works as expected in automatic1111 txt2img; it does generate the guided content.

However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.

Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

[2blackbar]

what preprocessor we should use with tile controlnet model ? Using it without preprocessor gets "some" results but the resolution is kinda lower than if i would inpaint with 0.55 denoise, have to use cfg 2-3

[continue-revolution]

The inpainting model may need more considerations in implementation and perhaps we just get other models first.

I’m the author of sd-webui-segment-anything and I am planning connect my extension to your inpainting model.
So at this moment, the inpainting ControlNet cannot target at the mask only while not changing other parts, right?

my gradio demo does not have masked diffusion in it. what is displayed now is just original results from standard non-masked diffusion. but masked diffusion will be better.

Do you think there is a need to wait for an update of this extension? Is the current extension compatible with the new models, especially the inpainting model?

[lllyasviel]

The model works as expected in automatic1111 txt2img; it does generate the guided content.

However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.

Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

[pkuliyi2015]

The model works as expected in automatic1111 txt2img; it does generate the guided content.
However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.
Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

The right one. I must have done something wrong. But until now I cannot fix it.

[ProGamerGov]

Is there a PR in this repo yet for implementing ControlNet v1.1?

[lllyasviel]

The model works as expected in automatic1111 txt2img; it does generate the guided content.
However, as I directly download the model and use it in this extension, it produces severe artifacts. I read the source code for a while but still not clear what should be done to make it work.
Some initial observations:

• Severe ghost shadows and duplicated contours, regardless of tile overlaps
• Faded colors in txt2img (even if with 840000 VAEs)
• Has no effect when using noise inversion (maybe this is my code flaws; I'm checking it).

See here for one result: https://imgsli.com/MTY5ODQw

which one is cn11tile? left or right?

The right one. I must have done something wrong. But until now I cannot fix it.

from the result it looks like your input image is bigger than h/8 w/8.

for example, if you diffuse at 512 512, your tile need to be 64 64 and then use 3 cv2. pyrup to interpolate to 512.

or you can add a gaussion blur to the inputs to make it smoother

[lllyasviel]

Hi I have a recommended list of updates:

Control Model:
Implement the global average pooling before injection – read "global_average_pooling" item in the yaml file.

Depth:
Rename “depth” to “depth_midas”
“depth_leres” is already good
Add “depth_zoe”

Normal:
Add “normal_bae”
Remove previous “normal” (or rename it to “normal_midas”)

Canny/MLSD:
already good

Scribble:
Rename “fake_scribble” to “scribble_hed”
Add “scribble_pidi”
Remove “scribble” (it seems that this one is just binarize, sounds confusing, or just "threshold"?)

SoftEdge:
Rename “HED” to “softedge_hed”
Add “softedge_pidi”
Add “softedge_hedsafe”, “softedge_pidisafe”
Rename “pidinet” to “sketch_t2iadapter”

Segmentation:
Rename “seg” to “seg_ufade20K”
Add “seg_ofade20K ” and “seg_ofcoco”

Openpose:
“openpose” is good
Remove “openpose_hand”
Add “openpose_full”

Lineart:
Add “lineart”
Add “lineart_coarse”
Add “lineart_anime”

Shuffle:
Add “shuffle”

What do you think?

[catboxanon]

That list looks good to me.

Are the instructpix2pix and inpainting models already working out of the box? The former seemed to work but I also felt like it gave me mixed results, but I wasn't going to judge the quality yet, not knowing if it's missing something. Inpainting model I haven't tried yet. Tile model I assume would come a bit later since the model itself is in an unfinished state currently.

[CCRcmcpe]

PR WIP at #742.

[CCRcmcpe]

Recently renaming of annotators caused some downstream developers unhappy. We can implement renamings as display name change instead of ID change which causes API breaks.

Also on naming, the annotator name should imply which cnet model should be used and vice versa.

[lllyasviel]

i have an idea. what about adding some descriptions to yaml file of each cnet like "xxx_canny.yaml" has a "desc: this model needs canny preprocessor" and show it to gradio ui?

[CCRcmcpe]

The gradio part seems less than ideal. List items cannot show hover infos, at least, I tried the the DDIM sampler item in WebUI, it doesn't, however if you select it and hover on the selection box it shows.

[lllyasviel]

i mean like adding a gradio.label or something and show some desc text from model yaml after a model is loaded.
besides i think for api it is ok to have alias names.

[lllyasviel]

if u think it is ok i will begin to work on all 14 yaml files

[CCRcmcpe]

What about the old cnets (prior to 1.1)? They have no isolated yamls. I think it's better to implement it at code level, which is also localization friendly. I will wait response from repo owner.

[lllyasviel]

old cnets can just use blank text. we can only show texts when desc is avaliable

[lllyasviel]

@Mikubill why invert always binarize images?

[lllyasviel]

now I have to invert outside using photoshop on my own to use the lineart model

[CCRcmcpe]

That is a known issue, will be fixed.

[halr9000]

Would be awesome to auto select the most likely model after preproc is selected & vice versa. Won't prevent people from changing, but will save a needed step 90% of the time.

…

On Thu, Apr 13, 2023 at 3:30 PM lllyasviel ***@***.***> wrote: i have an idea. what about adding some descriptions to yaml file of each cnet like "xxx_canny.yaml" has a "desc: this model needs canny preprocessor" and show it to gradio ui? — Reply to this email directly, view it on GitHub <#736 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAEQMXRITOKXTD7KL6AY53XBBH4FANCNFSM6AAAAAAW4RNXZE> . You are receiving this because you commented.Message ID: ***@***.***>

[2blackbar]

that anime colorise works nice even at 1080x1080 res and it works not only for anime stuff but works best with animelike models , this is 768

[ljleb]

Although it is not official, temporalnet also uses none as preprocessor.

[marcsyp]

What is temporalnet, and is there any documentation for using it with a1111 + controlnet? Sounds interesting...

[ljleb]

Here is their hugging face: https://huggingface.co/CiaraRowles/TemporalNet. You can see an example of output on the linked twitter post: https://twitter.com/CiaraRowles1/status/1637486561917906944

It is not perfect, but it helps a lot with getting consistent results. It worked best with a lower weight in my limited testing. To use it with this extension, you must put controlnet in batch mode and enable the checkbox "loopback" at the bottom of the unit that has temporalnet as the selected model. What this does is, it will automatically feed the latest generated image of the batch back into this unit for each iteration. The first image of the batch for this unit is either the first image of the selected directory or the uploaded image.

[lllyasviel]

hello everyone, the finished tile model is released #1033

[Jonseed]

Can hires fix be used with ControlNet on just the second pass of hires fix? I don't have an input image for the first pass. I want ControlNet to use the first low-res generated image as the input image to help compose the hires second pass image. If I don't put in an input image at the start, it says there is no input image, and doesn't use ControlNet at all.

[bropines]

Can hires fix be used with ControlNet on just the second pass of hires fix? I don't have an input image for the first pass. I want ControlNet to use the first low-res generated image as the input image to help compose the hires second pass image. If I don't put in an input image at the start, it says there is no input image, and doesn't use ControlNet at all.

What prevents you from simply generating and transferring the photo to the controlnet area?

[Jonseed]

What prevents you from simply generating and transferring the photo to the controlnet area?

I don't want ControlNet to use the low-res image on the first pass, which I want SD to generate purely from the text prompt, just on the second pass when it makes the image hi-res (to help control that second pass). Typically if you use a high denoising in hires fix, then the composition is lost in the second pass. ControlNet could be used to help maintain composition on that second pass, even at higher denoising strengths (the new Tile model could be very helpful here to refine details on the second hires pass).

[catboxanon]

You can use ControlNet in img2img then. Hires fix is simply an img2img shortcut.

[Jonseed]

You can use ControlNet in img2img then. Hires fix is simply an img2img shortcut.

Yes. But then it makes it a two-step process to make a hires image from a text prompt. The purpose of hires fix is to automate that and make it a one-step process to generate a hires image in txt2img. It seems like if you don't have an input image to begin with, ControlNet should just use the first pass low-res image generated as the control image.

[lllyasviel]

Hello everyone, we have added lineart_anime_denoise preprocessor

it can remove noise and manga screentone from inputs

[ccachx]

I don't know what happened, I downloaded all the models below, but only a few preprocessors in webui can preview normally. https://huggingface.co/lllyasviel/ControlNet-v1-1/tree/main And the controlnet I am using is a ZIP file downloaded from below. https://github.com/Mikubill/sd-webui-controlnet

I noticed that the console returned error codes.
Traceback (most recent call last): File "urllib\request.py", line 1348, in do_open File "http\client.py", line 1282, in request File "http\client.py", line 1328, in _send_request File "http\client.py", line 1277, in endheaders File "http\client.py", line 1037, in _send_output File "http\client.py", line 975, in send File "http\client.py", line 1454, in connect File "ssl.py", line 513, in wrap_socket File "ssl.py", line 1071, in _create File "ssl.py", line 1342, in do_handshake ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "F:\AI\novelai-webui\py310\lib\site-packages\gradio\routes.py", line 394, in run_predict output = await app.get_blocks().process_api( File "F:\AI\novelai-webui\py310\lib\site-packages\gradio\blocks.py", line 1075, in process_api result = await self.call_function( File "F:\AI\novelai-webui\py310\lib\site-packages\gradio\blocks.py", line 884, in call_function prediction = await anyio.to_thread.run_sync( File "F:\AI\novelai-webui\py310\lib\site-packages\anyio\to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "F:\AI\novelai-webui\py310\lib\site-packages\anyio_backends_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "F:\AI\novelai-webui\py310\lib\site-packages\anyio_backends_asyncio.py", line 867, in run result = context.run(func, *args) File "F:\AI\novelai-webui\extensions\sd-webui-controlnet\scripts\controlnet.py", line 454, in run_annotator result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b) File "F:\AI\novelai-webui\extensions\sd-webui-controlnet\scripts\processor.py", line 192, in openpose result = model_openpose(img) File "F:\AI\novelai-webui\extensions\sd-webui-controlnet\annotator\openpose__init__.py", line 75, in **call** self.load_model() File "F:\AI\novelai-webui\extensions\sd-webui-controlnet\annotator\openpose__init__.py", line 53, in load_model load_file_from_url(body_model_path, model_dir=self.model_dir) File "F:\AI\novelai-webui\py310\lib\site-packages\basicsr\utils\download_util.py", line 98, in load_file_from_url download_url_to_file(url, cached_file, hash_prefix=None, progress=progress) File "F:\AI\novelai-webui\py310\lib\site-packages\torch\hub.py", line 597, in download_url_to_file u = urlopen(req) File "urllib\request.py", line 216, in urlopen File "urllib\request.py", line 519, in open File "urllib\request.py", line 536, in _open File "urllib\request.py", line 496, in _call_chain File "urllib\request.py", line 1391, in https_open File "urllib\request.py", line 1351, in do_open urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)>

Same problem？Has it been solved？

I have met this problem in mac os, and I solved if by installing the "Certificates.command" in the python application

[lllyasviel]

if you are in China and has SSL: CERTIFICATE_VERIFY_FAILED, please turn off the proxy. You can directly connect huggingface without proxy in China

[bropines]

But what if the annotator creates artifacts during preprocessing?

My lineart	Preprocessor

Edit:
This happens when:

1. If the picture has no background (i.e. png with 100% alpha channel)
2. If a picture with a fully cast color or black

Picture(sorry, but the project under nda, had to hide the megumin)	Preprocessor	Result

@lllyasviel Can this be fixed?

[lllyasviel]

hi everyone we improved preview window in
#1106

[theboylzh]

After updated the controlnet1.1, I found most preprocessor do not work. In my work flow, the openpose is the most important and when I use it, the preview image shows error and openpose dose not work at all(the new v1.1 model has download). And the code feedbacks as it:
Preview Resolution = 512 Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/gradio/routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 1015, in process_api result = await self.call_function( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "/root/miniconda3/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 542, in run_annotator result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b, json_pose_callback=json_acceptor.accept) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 208, in run_model from annotator.openpose import OpenposeDetector File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/__init__.py", line 16, in <module> from . import util File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/util.py", line 7, in <module> from .body import BodyResult, Keypoint File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 22, in <module> class BodyResult(NamedTuple): File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 23, in BodyResult keypoints: List[Keypoint | None] TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

[lllyasviel]

After updated the controlnet1.1, I found most preprocessor do not work. In my work flow, the openpose is the most important and when I use it, the preview image shows error and openpose dose not work at all(the new v1.1 model has download). And the code feedbacks as it: Preview Resolution = 512 Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/gradio/routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 1015, in process_api result = await self.call_function( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "/root/miniconda3/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 542, in run_annotator result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b, json_pose_callback=json_acceptor.accept) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 208, in run_model from annotator.openpose import OpenposeDetector File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/__init__.py", line 16, in <module> from . import util File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/util.py", line 7, in <module> from .body import BodyResult, Keypoint File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 22, in <module> class BodyResult(NamedTuple): File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 23, in BodyResult keypoints: List[Keypoint | None] TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

@huchenlei

[lllyasviel]

@theboylzh seems fixed in 1.1.128, try update and restart completely
see also f715763

[yamitzky]

@theboylzh Upgrading python version to 3.10+ may help you!

[huchenlei]

After updated the controlnet1.1, I found most preprocessor do not work. In my work flow, the openpose is the most important and when I use it, the preview image shows error and openpose dose not work at all(the new v1.1 model has download). And the code feedbacks as it: Preview Resolution = 512 Traceback (most recent call last): File "/root/miniconda3/lib/python3.8/site-packages/gradio/routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 1015, in process_api result = await self.call_function( File "/root/miniconda3/lib/python3.8/site-packages/gradio/blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "/root/miniconda3/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/root/miniconda3/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 542, in run_annotator result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b, json_pose_callback=json_acceptor.accept) File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 208, in run_model from annotator.openpose import OpenposeDetector File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/__init__.py", line 16, in <module> from . import util File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/util.py", line 7, in <module> from .body import BodyResult, Keypoint File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 22, in <module> class BodyResult(NamedTuple): File "/root/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/openpose/body.py", line 23, in BodyResult keypoints: List[Keypoint | None] TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

See PEP 604. The pipe operator is introduced in Python 3.10. Please confirm that you are using the correct python version.

@theboylzh

[continue-revolution]

@huchenlei
Could you remove this operator, like using List[Union[int, str]] instead of List[int | str]?
I don’t think this operator is functionally necessary, it is just making your code more human-readable. It is not worth updating the whole virtual environment for a new operator which is easily substitutable. For people who have a bad networking condition like me, downloading a new PyTorch takes 20 minutes, not to say all such packages. If you are not willing to do such change. I can make a PR a bit later.

[lllyasviel]

hi everyone, a bit late but we added depth leres++

see comparisons here:
#1141

[shenmxd0762]

I want to change the yellow curtains in an anime room to green, but the inpaint feature makes the left and right curtain textures inconsistent. There are still noticeable details differences in enabling Canny. A similar requirement is to "add a few trees to the background", but the trees generated in each area need to be very similar. I don't know if it is possible to take advantage of controlnet to limit correlation between regions.

#968

I want to use inpaint to make both ears similar. Her right ear is made in t2i through prompt: cat_ girl, anthro_female_cat, you can find that the left ear is a bad ear. I used the same settings as yours, but it seems to have gotten worse.
Is this difference due to the different prompts used in the left and right ears? And is it possible to solve this problem by controlnet in the future?
Here is my prompt.

A similar requirement is that I want to change the plant on the right to the same variety as the one on the left.

Using Photoshop's mirroring function can solve some problems, but the light and shadow will become discontinuous.

[herval]

It seems batch_size is now being ignored by control net (using 1.1.137) - no matter the number, it always outputs one single image

[ljleb]

I cannot reproduce, using 1 unit as leres++ depth. Can you share your settings and other extensions? Also, you may want to create an issue instead of reporting an error here to help us keep track of the resolution.

[lllyasviel]

we will close this issue because it is too long. please use Discussion or Issue to report problems or discuss features

[lllyasviel]

hi everyone we released a discussion about tile model's color offset problem with anime images/models
#1164

[lllyasviel]

hi everyone we released a [Major Update] Reference-only Control
#1237