get bad result by StableDiffusionControlNetInpaintPipeline

[MMY1994]

webui result:

diffusers result:

Why did the human face change, I did not mask the face part.

This is my code:

def make_inpaint_condition(image, image_mask):
image = np.array(image.convert("RGB")).astype(np.float32) / 255.0
image_mask = np.array(image_mask.convert("L")).astype(np.float32) / 255.0

assert image.shape[0:1] == image_mask.shape[0:1]
image[image_mask > 0.5] = -1.0  # set as masked pixel
image = np.expand_dims(image, 0).transpose(0, 3, 1, 2)
image = torch.from_numpy(image)
return image

init_image = load_image("./init_image.jpg")
init_image = init_image.resize((512, 512))

mask_image = load_image("./mask_image.jpg")
mask_image = mask_image.resize((512, 512))

control_image = make_inpaint_condition(init_image, mask_image)

controlnet = ControlNetModel.from_pretrained("runwayml/stable-diffusion-v1-5", torch_dtype=torch.float16)

pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5", controlnet=controlnet, torch_dtype=torch.float16, use_safetensors=True
)
pipe.load_lora_weights("/cloudfs-data/visiondata/mmy/diffusers", weight_name="sd_lora_lbx_1e4.safetensors")
pipe.fuse_lora(lora_scale=0.9)

pipe.scheduler = EulerAncestralDiscreteScheduler.from_config(pipe.scheduler.config)

pipe.enable_model_cpu_offload()
pipe.enable_xformers_memory_efficient_attention()
pipe.to("cuda:3")

prompt = "a man is wearing brown safety shoes with multiple laces and a rugged sole"

negative_prompt="cartoon, painting, deformed, illustration, (worst quality, low quality, normal quality:2), (disconnected limbs:1.5), busty, (epicnegative:0.9), nsfw, (extra limb:1.5), (mutated hands and fingers), (poorly drawn hands)"

output = pipe(
prompt,
width=512,
height=512,
num_inference_steps=25,
negative_prompt=negative_prompt,
eta=1.0,
image=init_image,
mask_image=mask_image,
control_image=control_image,
).images[0]

[tolgacangoz]

Could you try with lllyasviel/control_v11p_sd15_inpaint controlnet?
Also, could you remove pipe.to("cuda:3")? It is not proper when using .enable_model_cpu_offload(). Just set pipe.enable_model_cpu_offload(gpu_id=3).

[MMY1994]

Sorry, I write a mistake.
I have used lllyasviel/control_v11p_sd15_inpaint. All models I use are official models except lora. But this problem still here even I don't use lora
controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_inpaint", torch_dtype=torch.float16)

I try to set pipe.enable_model_cpu_offload(gpu_id=3). But it still remain. I can not understand why only face have changed

[tolgacangoz]

Two recommendations:

1. Try resolution 1024x1024 for this setup when inferencing. But 2. one could be better.
2. Use StableDiffusionXLControlNetInpaintPipeline.

[MMY1994]

I use "runwayml/stable-diffusion-v1-5"at webui.

you mean I use another controlnet model match SDXL? Thank You!!! I will try it.
But I still do not understand why only face have changed. the same code I did not see somebody has this problem

[tolgacangoz]

Just try 1. suggestion with this setup, it helps. To fairly compare, StableDiffusionXLControlNetInpaintPipeline would be proper.

[MMY1994]

1024x1024 it really works!!!!, But Why?