Add Flux inpainting and Flux Img2Img #9135

Gothos · 2024-08-09T06:17:01Z

What does this PR do?

PR to add

flux inpainting.
flux img2img

Before submitting

Did you read the contributor guideline?
Did you read our philosophy doc (important for complex PRs)?
Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Adds basic flux inpainting. This still has some ways to go, especially since any flux equivalent of 9-channel inpainting is not supported yet. I'd also like comments on noising.
Image, mask, and inpainting a cactus at strengths from 0.65 to 0.9

a-r-r-o-w · 2024-08-12T15:01:11Z

@Gothos This is looking great! Since this PR is not yet marked for review, I assume it is incomplete in some ways. Let us know if you're facing any problems and we'd be happy to help. There are a couple of issues and messages from folks asking to have this implemented and usable from diffusers, so really nice of you to take this up :)

cc @asomoza here for more testing and implementation/noising improvements 🤩

Gothos · 2024-08-12T15:07:49Z

It works ootb. I probably should have marked it as ready for review really, since it's missing only the inpainting-only checkpoint (i.e models similar to stable-diffusion-xl-inpainting-0.1, which we don't have for flux) support and docs.

SkalskiP · 2024-08-12T17:49:40Z

Hi @Gothos 👋🏻 can you provide any usage example showing how to run the inpainting pipeline?

Gothos · 2024-08-12T18:42:21Z

Sure!

First

pip3 install git+https://github.com/Gothos/diffusers.git@flux-inpaint

then:

from diffusers import FluxInpaintPipeline
from PIL import Image
import torch

pipe =  FluxInpaintPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)

prompt = "your prompt here"
image = pipe(
    prompt,
    image = Image.open("path/to/image",
    mask_image=Image.open("path/to/mask"),
    strength=0.85, # below 0.85 doesn't seem to cause a lot of change
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image

Just replacing the path to the image and path to the mask, and prompt should work.

Gothos · 2024-08-13T05:41:19Z

@asomoza if I'm not wrong the inpainting trained flux endpoint should check for 132 channels? If this is the case I'll probably finish the PR today.

Gothos · 2024-08-13T06:32:20Z

Also correct me if I'm wrong, but isn't the img2img equivalent to having an all-white mask in inpainting/not selectively blending latents in denoise step? I can add in an img2img pipeline as well if this is the case, since it'll involve minimal changes from inpainting. @a-r-r-o-w @asomoza

Gothos · 2024-08-13T11:04:01Z

I've added in img2img as well now.
Image and img2img into a night scene, with strengths 0.65,0.7,0.75,0.8,0.85,0.9,0.95.

SkalskiP · 2024-08-13T13:50:18Z

@Gothos awesome work! I build FLUX.1 inpainting HF space using code from this PR: https://huggingface.co/spaces/SkalskiP/FLUX.1-inpaint

Gothos · 2024-08-13T13:52:27Z

Yeah saw the space and the linkedin post! Thanks for the mention!

DN6 · 2024-08-14T04:39:26Z

Nice work @Gothos! A few things before we merge. Can we

Resolve the merge conflicts
Update the PR description/title to also include Img2Img
I think we can remove the check for the 132 channels in the inpainting pipeline for now. The assumption here is reasonable, but since there isn't an actual checkpoint to test with, we don't need to preemptively add.
Can we add fast/slow tests for the pipelines.

Gothos · 2024-08-14T04:40:44Z

Will do today.

Gothos · 2024-08-14T04:58:05Z

Do you also suggest I put in #9153 for these two pipelines, @DN6 ?

DN6 · 2024-08-14T05:14:22Z

@Gothos Yeah you can do that as well 👍🏽

Gothos · 2024-08-14T05:15:31Z

Cool, will do all these and request a review.

fursund · 2024-08-14T07:30:59Z

Looks like this fails on mac with MPS. There's been some recent fixes to FLUX for diffusers that might have to be added here as well?

fursund · 2024-08-14T07:32:33Z

This is the error I get: TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.

Gothos · 2024-08-14T07:32:48Z

Hmm I don't really have a mac to test this. Could you point out the PR?

fursund · 2024-08-14T07:36:28Z

Hmm maybe it's still broken: #9047 ... potentially this fix: #9097

Gothos · 2024-08-14T07:40:46Z

Hmm maybe it's still broken: #9047 ... potentially this fix: #9097

It still is, for most torch dists I guess. Try torch 2.4 or above. It might fix this.

yiyixuxu · 2024-09-03T03:55:50Z

src/diffusers/pipelines/flux/pipeline_flux_img2img.py

+        shape = (batch_size, num_channels_latents, height, width)
+        latent_image_ids = self._prepare_latent_image_ids(batch_size, height, width, device, dtype)
+
+        if latents is not None:


to be consistent with the defination of latents input in our other img2img pipelines (they are image latents)

diffusers/src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl_img2img.py

Line 1277 in 007ad0e

if latents is None:

yiyixuxu · 2024-09-03T06:14:01Z

src/diffusers/pipelines/flux/pipeline_flux_inpaint.py

+
+        if latents is None:
+            noise = randn_tensor(shape, generator=generator, device=device, dtype=dtype)
+            latents = self.scheduler.scale_noise(image_latents, timestep, noise)


note that we do not need is_strength_max for flow match based models: it is a pure noise when strengh==1

diffusers/src/diffusers/schedulers/scheduling_flow_match_euler_discrete.py

Line 161 in 1c1ccaa

sample = sigma * noise + (1.0 - sigma) * sample

will remove that for sd3 inpaint too

yiyixuxu · 2024-09-03T08:24:50Z

@Gothos
thanks for your PR!
I made some final changes. We will merge this very soon.

If you can make some final checks, that would be great! (no worries if not)

nd sorry we're a bit slow in this

Gothos · 2024-09-03T08:26:51Z

Haha I should be the one apologising, I've been too slow on this! I'll run some examples on my end.

…

On Tue, 3 Sept 2024, 13:55 YiYi Xu, ***@***.***> wrote: @Gothos <https://github.com/Gothos> thanks for your PR! I made some final changes. We will merge this very soon. If you can make some final checks, that would be great! (no worries if not) nd sorry we're a bit slow in this — Reply to this email directly, view it on GitHub <#9135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWY3A7NHSVNTQ45TGLUTEDLZUVW6VAVCNFSM6AAAAABMH2M4LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRVHEYDANJYGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

--------- Co-authored-by: yiyixuxu <[email protected]> Update `UNet2DConditionModel`'s error messages (#9230) * refactor [CI] Update Single file Nightly Tests (#9357) * update * update feedback. improve README for flux dreambooth lora (#9290) * improve readme * improve readme * improve readme * improve readme fix one uncaught deprecation warning for accessing vae_latent_channels in VaeImagePreprocessor (#9372) deprecation warning vae_latent_channels add mixed int8 tests and more tests to nf4. [core] Freenoise memory improvements (#9262) * update * implement prompt interpolation * make style * resnet memory optimizations * more memory optimizations; todo: refactor * update * update animatediff controlnet with latest changes * refactor chunked inference changes * remove print statements * update * chunk -> split * remove changes from incorrect conflict resolution * remove changes from incorrect conflict resolution * add explanation of SplitInferenceModule * update docs * Revert "update docs" This reverts commit c55a50a. * update docstring for freenoise split inference * apply suggestions from review * add tests * apply suggestions from review quantization docs. docs.

This reverts commit 5799954.

ukaprch · 2024-09-23T12:10:26Z

What I can tell you is that as good as Flux is for modest inpainting (filling in a masked region) it is very poor at outpainting (replacing everything but the mask object). Flux needs an inpainting version.

ssxxx1a · 2024-09-23T12:15:30Z

@ssxxx1a try higher denoising strength. Larger than 0.85 works fine, start from 1. to understand if that is the issue.

it will lose the ability of inpainting. as a text2img task in designated area with mask

ukaprch · 2024-09-26T20:54:14Z

When I use higher strength FLUX overpaints a new image with my masked image. It doesn't scale or blend properly as inpainting should. Yes, if I reduce strength it will somewhat work but not as well as an inpainting model should. The whole point is that FLUX needs an Inpainting model to work as well as SDXL inpainting works.

…

On Mon, Sep 23, 2024 at 8:15 AM ssxxx1a ***@***.***> wrote: @ssxxx1a <https://github.com/ssxxx1a> try higher denoising strength. Larger than 0.85 works fine, start from 1. to understand if that is the issue. it will lose the ability of inpainting. as a text2img task in designated area with mask — Reply to this email directly, view it on GitHub <#9135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZTE5IBJQWLKQ5EFAL3U7HDZYAA73AVCNFSM6AAAAABMH2M4LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRYGA2DGNJTGI> . You are receiving this because you commented.Message ID: ***@***.***>

pandayummy · 2024-09-27T01:27:50Z

We need an Flux inpanting model like this:
https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1

But it requirs a lot of GPU resources/money to train.

ukaprch · 2024-09-30T16:44:32Z

Unfortunately yes if FLUX is to do proper inpainting / outpainting just like SDXL does.

…

On Thu, Sep 26, 2024 at 9:28 PM PandaYummy ***@***.***> wrote: We need an Flux inpanting model like this: https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1 But it requirs a lot of GPU resources/money to train. — Reply to this email directly, view it on GitHub <#9135 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZTE5IET6IV5IIPOLHOS5SDZYSYC7AVCNFSM6AAAAABMH2M4LGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZYGIZDQMJRGQ> . You are receiving this because you commented.Message ID: ***@***.***>

* quantization config. * fix-copies * fix * modules_to_not_convert * add bitsandbytes utilities. * make progress. * fixes * quality * up * up rotary embedding refactor 2: update comments, fix dtype for use_real=False (#9312) fix notes and dtype up up * minor * up * up * fix * provide credits where due. * make configurations work. * fixes * fix * update_missing_keys * fix * fix * make it work. * fix * provide credits to transformers. * empty commit * handle to() better. * tests * change to bnb from bitsandbytes * fix tests fix slow quality tests SD3 remark fix complete int4 tests add a readme to the test files. add model cpu offload tests warning test * better safeguard. * change merging status * courtesy to transformers. * move upper. * better * make the unused kwargs warning friendlier. * harmonize changes with huggingface/transformers#33122 * style * trainin tests * feedback part i. * Add Flux inpainting and Flux Img2Img (#9135) --------- Co-authored-by: yiyixuxu <[email protected]> Update `UNet2DConditionModel`'s error messages (#9230) * refactor [CI] Update Single file Nightly Tests (#9357) * update * update feedback. improve README for flux dreambooth lora (#9290) * improve readme * improve readme * improve readme * improve readme fix one uncaught deprecation warning for accessing vae_latent_channels in VaeImagePreprocessor (#9372) deprecation warning vae_latent_channels add mixed int8 tests and more tests to nf4. [core] Freenoise memory improvements (#9262) * update * implement prompt interpolation * make style * resnet memory optimizations * more memory optimizations; todo: refactor * update * update animatediff controlnet with latest changes * refactor chunked inference changes * remove print statements * update * chunk -> split * remove changes from incorrect conflict resolution * remove changes from incorrect conflict resolution * add explanation of SplitInferenceModule * update docs * Revert "update docs" This reverts commit c55a50a. * update docstring for freenoise split inference * apply suggestions from review * add tests * apply suggestions from review quantization docs. docs. * Revert "Add Flux inpainting and Flux Img2Img (#9135)" This reverts commit 5799954. * tests * don * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * contribution guide. * changes * empty * fix tests * harmonize with huggingface/transformers#33546. * numpy_cosine_distance * config_dict modification. * remove if config comment. * note for load_state_dict changes. * float8 check. * quantizer. * raise an error for non-True low_cpu_mem_usage values when using quant. * low_cpu_mem_usage shenanigans when using fp32 modules. * don't re-assign _pre_quantization_type. * make comments clear. * remove comments. * handle mixed types better when moving to cpu. * add tests to check if we're throwing warning rightly. * better check. * fix 8bit test_quality. * handle dtype more robustly. * better message when keep_in_fp32_modules. * handle dtype casting. * fix dtype checks in pipeline. * fix warning message. * Update src/diffusers/models/modeling_utils.py Co-authored-by: YiYi Xu <[email protected]> * mitigate the confusing cpu warning --------- Co-authored-by: Vishnu V Jaddipal <[email protected]> Co-authored-by: Steven Liu <[email protected]> Co-authored-by: YiYi Xu <[email protected]>

Nomination-NRB · 2024-10-23T01:36:46Z

Sure!

First

pip3 install git+https://github.com/Gothos/diffusers.git@flux-inpaint

then:

from diffusers import FluxInpaintPipeline
from PIL import Image
import torch

pipe =  FluxInpaintPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)

prompt = "your prompt here"
image = pipe(
    prompt,
    image = Image.open("path/to/image",
    mask_image=Image.open("path/to/mask"),
    strength=0.85, # below 0.85 doesn't seem to cause a lot of change
    height=1024,
    width=1024,
    guidance_scale=3.5,
    num_inference_steps=50,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image

Just replacing the path to the image and path to the mask, and prompt should work.

Thanks for your code, and how much GPU VRAM consume?

--------- Co-authored-by: yiyixuxu <[email protected]>

* quantization config. * fix-copies * fix * modules_to_not_convert * add bitsandbytes utilities. * make progress. * fixes * quality * up * up rotary embedding refactor 2: update comments, fix dtype for use_real=False (#9312) fix notes and dtype up up * minor * up * up * fix * provide credits where due. * make configurations work. * fixes * fix * update_missing_keys * fix * fix * make it work. * fix * provide credits to transformers. * empty commit * handle to() better. * tests * change to bnb from bitsandbytes * fix tests fix slow quality tests SD3 remark fix complete int4 tests add a readme to the test files. add model cpu offload tests warning test * better safeguard. * change merging status * courtesy to transformers. * move upper. * better * make the unused kwargs warning friendlier. * harmonize changes with huggingface/transformers#33122 * style * trainin tests * feedback part i. * Add Flux inpainting and Flux Img2Img (#9135) --------- Co-authored-by: yiyixuxu <[email protected]> Update `UNet2DConditionModel`'s error messages (#9230) * refactor [CI] Update Single file Nightly Tests (#9357) * update * update feedback. improve README for flux dreambooth lora (#9290) * improve readme * improve readme * improve readme * improve readme fix one uncaught deprecation warning for accessing vae_latent_channels in VaeImagePreprocessor (#9372) deprecation warning vae_latent_channels add mixed int8 tests and more tests to nf4. [core] Freenoise memory improvements (#9262) * update * implement prompt interpolation * make style * resnet memory optimizations * more memory optimizations; todo: refactor * update * update animatediff controlnet with latest changes * refactor chunked inference changes * remove print statements * update * chunk -> split * remove changes from incorrect conflict resolution * remove changes from incorrect conflict resolution * add explanation of SplitInferenceModule * update docs * Revert "update docs" This reverts commit c55a50a. * update docstring for freenoise split inference * apply suggestions from review * add tests * apply suggestions from review quantization docs. docs. * Revert "Add Flux inpainting and Flux Img2Img (#9135)" This reverts commit 5799954. * tests * don * Apply suggestions from code review Co-authored-by: Steven Liu <[email protected]> * contribution guide. * changes * empty * fix tests * harmonize with huggingface/transformers#33546. * numpy_cosine_distance * config_dict modification. * remove if config comment. * note for load_state_dict changes. * float8 check. * quantizer. * raise an error for non-True low_cpu_mem_usage values when using quant. * low_cpu_mem_usage shenanigans when using fp32 modules. * don't re-assign _pre_quantization_type. * make comments clear. * remove comments. * handle mixed types better when moving to cpu. * add tests to check if we're throwing warning rightly. * better check. * fix 8bit test_quality. * handle dtype more robustly. * better message when keep_in_fp32_modules. * handle dtype casting. * fix dtype checks in pipeline. * fix warning message. * Update src/diffusers/models/modeling_utils.py Co-authored-by: YiYi Xu <[email protected]> * mitigate the confusing cpu warning --------- Co-authored-by: Vishnu V Jaddipal <[email protected]> Co-authored-by: Steven Liu <[email protected]> Co-authored-by: YiYi Xu <[email protected]>

Gothos and others added 5 commits August 2, 2024 07:29

Fix from_single_file for xl_inpaint

28469de

Merge branch 'huggingface:main' into main

7e21edf

Add basic flux inpaint pipeline

31d3707

style, quality, stray print

e305a0a

Fix stray changes

2f58003

Gothos mentioned this pull request Aug 12, 2024

Add Flux inpainting pipeline #9148

Closed

2 tasks

3d1b1c2

Add inpainting model support

7bbf12c

Gothos marked this pull request as ready for review August 13, 2024 12:06

a-r-r-o-w requested a review from asomoza August 13, 2024 12:07

Merge branch 'main' into flux-inpaint

b67a289

yiyixuxu added 4 commits September 3, 2024 05:35

update imgimg

2a06163

copies

b870eed

Merge remote-tracking branch 'origin' into flux-inpaint

85d0ab6

copies

7bd0d53

yiyixuxu reviewed Sep 3, 2024

View reviewed changes

yiyixuxu added 3 commits September 3, 2024 07:22

update inpaint

9616608

remove is_strength_max

844a12e

style

9796496

yiyixuxu reviewed Sep 3, 2024

View reviewed changes

yiyixuxu approved these changes Sep 3, 2024

View reviewed changes

yiyixuxu added 4 commits September 3, 2024 09:38

update test

9f21255

add doc

0d2f98c

format docstring example

028b0af

Merge branch 'main' into flux-inpaint

056ebe3

yiyixuxu merged commit 249a9e4 into huggingface:main Sep 4, 2024
14 of 15 checks passed

sayakpaul added a commit that referenced this pull request Sep 6, 2024

Revert "Add Flux inpainting and Flux Img2Img (#9135)"

8e4bd08

This reverts commit 5799954.

yiyixuxu mentioned this pull request Sep 12, 2024

[FLUX] add Img2Img pipeline #9070

Closed

5 tasks

waynemadsen mentioned this pull request Sep 12, 2024

FLUX Support Acly/krita-ai-diffusion#1152

Closed

sayakpaul pushed a commit that referenced this pull request Dec 23, 2024

Add Flux inpainting and Flux Img2Img (#9135)

2a72888

--------- Co-authored-by: yiyixuxu <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Flux inpainting and Flux Img2Img #9135

Add Flux inpainting and Flux Img2Img #9135

Gothos commented Aug 9, 2024 •

edited

Loading

a-r-r-o-w commented Aug 12, 2024 •

edited

Loading

Gothos commented Aug 12, 2024 •

edited

Loading

SkalskiP commented Aug 12, 2024

Gothos commented Aug 12, 2024 •

edited

Loading

Gothos commented Aug 13, 2024

Gothos commented Aug 13, 2024

Gothos commented Aug 13, 2024

SkalskiP commented Aug 13, 2024

Gothos commented Aug 13, 2024

DN6 commented Aug 14, 2024

Gothos commented Aug 14, 2024

Gothos commented Aug 14, 2024

DN6 commented Aug 14, 2024

Gothos commented Aug 14, 2024 •

edited

Loading

fursund commented Aug 14, 2024

fursund commented Aug 14, 2024

Gothos commented Aug 14, 2024

fursund commented Aug 14, 2024

Gothos commented Aug 14, 2024

yiyixuxu Sep 3, 2024

yiyixuxu Sep 3, 2024

yiyixuxu commented Sep 3, 2024

Gothos commented Sep 3, 2024 via email

ukaprch commented Sep 23, 2024

ssxxx1a commented Sep 23, 2024

ukaprch commented Sep 26, 2024 via email

pandayummy commented Sep 27, 2024

ukaprch commented Sep 30, 2024 via email

Nomination-NRB commented Oct 23, 2024

Add Flux inpainting and Flux Img2Img #9135

Add Flux inpainting and Flux Img2Img #9135

Conversation

Gothos commented Aug 9, 2024 • edited Loading

What does this PR do?

Before submitting

a-r-r-o-w commented Aug 12, 2024 • edited Loading

Gothos commented Aug 12, 2024 • edited Loading

SkalskiP commented Aug 12, 2024

Gothos commented Aug 12, 2024 • edited Loading

Gothos commented Aug 13, 2024

Gothos commented Aug 13, 2024

Gothos commented Aug 13, 2024

SkalskiP commented Aug 13, 2024

Gothos commented Aug 13, 2024

DN6 commented Aug 14, 2024

Gothos commented Aug 14, 2024

Gothos commented Aug 14, 2024

DN6 commented Aug 14, 2024

Gothos commented Aug 14, 2024 • edited Loading

fursund commented Aug 14, 2024

fursund commented Aug 14, 2024

Gothos commented Aug 14, 2024

fursund commented Aug 14, 2024

Gothos commented Aug 14, 2024

yiyixuxu Sep 3, 2024

Choose a reason for hiding this comment

yiyixuxu Sep 3, 2024

Choose a reason for hiding this comment

yiyixuxu commented Sep 3, 2024

Gothos commented Sep 3, 2024 via email

ukaprch commented Sep 23, 2024

ssxxx1a commented Sep 23, 2024

ukaprch commented Sep 26, 2024 via email

pandayummy commented Sep 27, 2024

ukaprch commented Sep 30, 2024 via email

Nomination-NRB commented Oct 23, 2024

Gothos commented Aug 9, 2024 •

edited

Loading

a-r-r-o-w commented Aug 12, 2024 •

edited

Loading

Gothos commented Aug 12, 2024 •

edited

Loading

Gothos commented Aug 12, 2024 •

edited

Loading

Gothos commented Aug 14, 2024 •

edited

Loading