flux controlnet control_guidance_start and control_guidance_end implement

[ighoshsubho]

What does this PR do?

Fixes #9555

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• [] Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@yiyixuxu

[asomoza]

thanks a lot, @simbrams can you give this a test and see if it improves your use case?

[simbrams]

Thanks a lot @ighoshsubho !

thanks a lot, @simbrams can you give this a test and see if it improves your use case?

I wil be trying it soon, will let you know about the results ASAP.

[ighoshsubho]

@simbrams any update on this fix, does it solve what you were looking for? And @yiyixuxu is there any more changes required, if not can this be merged?

[yiyixuxu]

@asomoza can you give this a review?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[asomoza]

LGTM overall, thanks a lot, left a couple of comments

[asomoza]

I took the opportunity to test the jasperai controlnet, here's the example with this applied and a controlnet_conditioning_scale of 1.0:

default	control_guidance_end=0.5	control_guidance_end=0.5 + control_guidance_start=0.2

[simbrams]

My tests with inpainting are not very conclusive unfortunately, I haven't been able to find parameters that works well with inpainting and "strong" guidance with Depth. If you guys have any tweak ideas that would work to achieve the result bellow, please let me know :)

Flux model used: black-forest-labs/FLUX.1-dev
Depth model used: jasperai/Flux.1-dev-Controlnet-Depth

Inputs:

Image	Mask	Depth	"Desired" Output
			Output using SD 1.5

Current results:

strength: 0.8
steps: 100
controlnet_conditioning_scale: 0.8
controlnet_guidance_start: 0
controlnet_guidance_end: 0.4

strength: 0.8
steps: 100
controlnet_conditioning_scale: 0.6 🔻
controlnet_guidance_start: 0
controlnet_guidance_end: 0.4

strength: 0.8
steps: 100
controlnet_conditioning_scale: 0.6
controlnet_guidance_start: 0
controlnet_guidance_end: 0.6 ↗️

strength: 0.8
steps: 100
controlnet_conditioning_scale: 0.4 🔻
controlnet_guidance_start: 0
controlnet_guidance_end: 0.8 ↗️

strength: 0.8
steps: 100
controlnet_conditioning_scale: 0.5 ↗️
controlnet_guidance_start: 0
controlnet_guidance_end: 0.8

[burgalon]

any chance to get this merged?

[asomoza]

@simbrams this just confirms what I was thinking and saw, Flux right now is not a good solution for inpainting, maybe it will get better with time but the model since it's distilled and very forceful aesthetic might be a problem with realistic photo inpainting.

You have a very little margin to work with it and it takes a long time to be able to experiment with it and even more to train a decent controlnet.

Maybe you can try with the recent "undistilled" models? I think there's five of them right now, they will take more time for inference since they have CFG but probably will yield better results.

[simbrams]

@simbrams this just confirms what I was thinking and saw, Flux right now is not a good solution for inpainting, maybe it will get better with time but the model since it's distilled and very forceful aesthetic might be a problem with realistic photo inpainting.

You have a very little margin to work with it and it takes a long time to be able to experiment with it and even more to train a decent controlnet.

Maybe you can try with the recent "undistilled" models? I think there's five of them right now, they will take more time for inference since they have CFG but probably will yield better results.

Indeed you were right ! I will try with undistilled models and see how it goes :) thanks for the tip!

UPDATE:
Just saw undistilled like nyanko7/flux-dev-de-distill is currently not compatible with diffusers unfortunately.

[asomoza]

oh I forgot about it, they need the CFG so they probably only work with the community pipeline and currently we don't have a controlnet pipeline for Flux with CFG.

You can open an issue about and we can discuss it there, I'll merge this PR now.

[asomoza]

Thanks a lot @ighoshsubho, the failing test is unrelated.