Loading LCM-Lora after or before loading ip-adapter?

[juancopi81]

Hi,

I was looking at the docs and found that it is better to load the ip-adapter weights before loading the LCM-lora:

"Note that you need to load IP-Adapter weights before loading the LCM-Lora weights."

pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipe.load_lora_weights(lcm_lora_id)

For my use case, it is better to have loaded the LCM-lora and at some point load the ip-adapter. I tried both loading after and before the ip-adapter and the results are the same (at least visually).

Is this expected?

[sayakpaul]

Thanks for initiating.

Which order of loading gives you the expected results?

[juancopi81]

Hi @sayakpaul and thank you very much for your answer. Loading the ip-adapter before and after loading the LCM-Lora seems to have no effect (they produce the same image). I was wondering if there is any reason why the docs recommend first loading the ip-adapter and the the LCM-Lora.

I found here:

"Note that you need to load IP-Adapter weights before loading the LCM-Lora weights."

I tested both scenarios following this code:

from diffusers import DiffusionPipeline, LCMScheduler
import torch
from diffusers.utils import load_image

model_id =  "sd-dreambooth-library/herge-style"
lcm_lora_id = "latent-consistency/lcm-lora-sdv1-5"

pipe = DiffusionPipeline.from_pretrained(model_id)

# For the first test
pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipe.load_lora_weights(lcm_lora_id)
pipe.scheduler = LCMScheduler.from_config(pipe.scheduler.config)
# For the second test
# pipe.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")

prompt = "best quality, high quality"
image = load_image("https://user-images.githubusercontent.com/24734142/266492875-2d50d223-8475-44f0-a7c6-08b51cb53572.png")

# Seed for reproductibility
generator = torch.Generator(device="cpu").manual_seed(32)
images = pipe(
    prompt=prompt,
    ip_adapter_image=image,
    num_inference_steps=4,
    guidance_scale=1,
    generator=generator
).images[0]

And both produce the same image:
Image in first scenario (loading the ip-adapter first):

Image in second scenario (loading the LCM_Lora first)

So I was wondering if there is any reason the docs recommend one approach over the other.

Thanks again!!

[sayakpaul]

I will let @patil-suraj answer further. Cc @yiyixuxu too for awareness.

[yiyixuxu]

We can maybe remove that line from the doc?

[juancopi81]

Thanks @yiyixuxu I did no notice any difference, but just wanted to ask if I am missing something :)

[skywalker00001]

There should be no difference. You can verify by substracting 1img from 2img, the result is supposed to be 0.