I want to use two controlnets in the api.

[xiaoxuwei12345]

I want to use two controlnets in the api. I used the following method, but the generated picture has a strange effect. Is there something wrong with my wording? Or is it api bug?

A1111 payload

payload = {
"prompt": '',
"negative_prompt": "",
"batch_size": 1,
"steps": 20,
"cfg_scale": 7,
"alwayson_scripts": {
"controlnet": {
"args": [
{
"input_image": encoded_image,
"module": "canny",
"model": "control_v11p_sd15_canny [d14c016b]",
},
{
"input_image": encoded_image2,
"module": "t2ia_style_clipvision",
"model": "t2iadapter_style_sd14v1 [202e85cc]",
}
]
}
}
}

[alfredo-diaz]

Do you also get a successful (i.e. status 200) output payload response from the API? If so, I seem to be a having similar problem with the img2img API endpoint. I am not getting the same results with the API as I would with the webUI while using the same settings/parameters in both cases.

From a few hours of testing, comparing results between the webui and the API (with the same settings/parameters), it seems to me that in my case, my image output seems to apply the img2img model/settings, but completely ignores and doesn't apply the ControlNet model/settings I specified in the payload (even though it confirms that it supposedly does so in the output payload in the "info" key/section of the output payload; the "info" section indicates "ControlNet 0", the canny model, and "ControlNet 1", the depth model, each with their correct respective settings indicated). Below you'll be able to find my input payload to the API, with the base64 string being replaced with an actual base64 string upon actually sending the input payload (to reiterate, I am able to get a response back from the API with this input payload, but it isn't the response I need or expected).

I apologize if I'm missing anything or interpreting something incorrectly as I'd definitely consider myself a "noob" and my technical coding knowledge is pretty limited, but any help is greatly appreciated!!! I've been scavenging around for answers with no success yet

{
"init_images": ["base64_image_string"],
"denoising_strength": 0.85,
"prompt": "modern disney style, a dog, same color as input image, contoured edges, cute eyes, black colored colored dog",
"steps": 20,
"width": 645,
"sampler_name": "Euler a",
"height": 1398,
"image_cfg_scale": 7,
"sd _model": "moDi-v1-pruned",
"restore_faces": false,
"sampler_index": "Euler",
"save_images": false,
"alwayson_scripts": {
"ControlNet": {
"args": [
{
"input_image": null,
"module": "canny",
"model": "control_v11p_sd15_canny.pth [d14c016b]",
"resize_mode": 1,
"weight": 1,
"lowvram": false,
"processor_res": 64,
"guidance_start": 0.0,
"guidance_end": 1.0,
"control_mode": 0,
"threshold_a": 9,
"threshold_b": 45
},
{
"input_image:"" null,
"module": "depth_zoe",
"resize_mode": 1,
"weight": 1,
"model": "control_v11f1p_sd15_depth.pth [cfd03158]",
"lowvram": false,
"processor_res": 64,
"guidance_start": 0.0,
"guidance_end": 1.0,
"control_mode": 0
}
]
}
}
}

[stepsal]

@alfredo-diaz did you get a solution to this problem. I have the same issue

[huchenlei]

You need to set "enabled": true in the control net args.

You can use this extension to display the API payload after WebUI generation:
https://github.com/huchenlei/sd-webui-api-payload-display

Here is my ControlNet payload that enables 2 units of lineart.

 "ControlNet": {
            "args": [
                {
                    "batch_images": "",
                    "control_mode": "ControlNet is more important",
                    "enabled": true,
                    "guidance_end": 1,
                    "guidance_start": 0,
                    "image": {
                        "image": "[base64img1]"
                    },
                    "input_mode": "simple",
                    "is_ui": true,
                    "loopback": false,
                    "low_vram": false,
                    "model": "control_v11p_sd15_lineart [43d4be0d]",
                    "module": "lineart_standard (from white bg & black line)",
                    "output_dir": "",
                    "pixel_perfect": false,
                    "processor_res": 512,
                    "resize_mode": "Resize and Fill",
                    "threshold_a": 64,
                    "threshold_b": 64,
                    "weight": 1
                },
                {
                    "batch_images": "",
                    "control_mode": "ControlNet is more important",
                    "enabled": true,
                    "guidance_end": 1,
                    "guidance_start": 0,
                    "image": {
                        "image": "[base64img2]",
                    },
                    "input_mode": "simple",
                    "is_ui": true,
                    "loopback": false,
                    "low_vram": false,
                    "model": "control_v11p_sd15_lineart [43d4be0d]",
                    "module": "lineart_standard (from white bg & black line)",
                    "output_dir": "",
                    "pixel_perfect": false,
                    "processor_res": 512,
                    "resize_mode": "Resize and Fill",
                    "threshold_a": 64,
                    "threshold_b": 64,
                    "weight": 1
                },
                {
                    "batch_images": "",
                    "control_mode": "Balanced",
                    "enabled": false,
                    "guidance_end": 1,
                    "guidance_start": 0,
                    "image": null,
                    "input_mode": "simple",
                    "is_ui": true,
                    "loopback": false,
                    "low_vram": false,
                    "model": "None",
                    "module": "none",
                    "output_dir": "",
                    "pixel_perfect": false,
                    "processor_res": 64,
                    "resize_mode": "Crop and Resize",
                    "threshold_a": 64,
                    "threshold_b": 64,
                    "weight": 1
                }
            ]
        },

[ljleb]

The ui to api payload tool looks very useful, thanks for making it.

I just wanted to mention the presence of some of the keys could be misleading, for example "batch_images", which the api does not accept. It's a ui only parameter. It will still work but the external_code.ControlNetUnit constructor will ignore these extraneous parameters.

[milindpattanaik]

I wanted to know how do you use this extension , should the script be placed in webui folder or it is run separately . How is it done . The read-me file doesn't say much