Feature flux controlnet img2img and inpaint pipeline

[ighoshsubho]

What does this PR do?

Fixes #9402 and #9158

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?
• Was this discussed/approved via a GitHub issue or the forum? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[ighoshsubho]

Hey @a-r-r-o-w can you suggest if the flux img2img controlnet pipeline needs any fix or if this architecture is fine to go with.

[yiyixuxu]

thanks!

[yiyixuxu]

awesome! thank you!
can you share example script + result from these two pipelines?

[yiyixuxu]

can you run make style and fix copies so CI can pass?

[ighoshsubho]

awesome! thank you!
can you share example script + result from these two pipelines?

Sure I will!

[ighoshsubho]

awesome! thank you! can you share example script + result from these two pipelines?

awesome! thank you! can you share example script + result from these two pipelines?

Here is example for img2img -

diffusers/src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py

Line 42 in a30ca65

Examples:

Here is example for inpaint -

diffusers/src/diffusers/pipelines/flux/pipeline_flux_controlnet_inpainting.py

Line 43 in a30ca65

Examples:

[ighoshsubho]

@yiyixuxu any updates, are we good to go or does it require further changes

[yiyixuxu]

can we look into the failing fast tests?

[a-r-r-o-w]

Thank you, the PR looks very clean and everything seems right! Great work @ighoshsubho

Just some minor nits from me. Could you also add these pipelines to the flux docs and address the failing tests?

Btw @yiyixuxu, do we need the slow integration tests here? I think we don't have them on derived pipelines that are made from two or more components (such as Flux + ImageToImage, or AnimateDiff + VideoToVideo)

[ighoshsubho]

Thank you, the PR looks very clean and everything seems right! Great work @ighoshsubho

Just some minor nits from me. Could you also add these pipelines to the flux docs and address the failing tests?

Btw @yiyixuxu, do we need the slow integration tests here? I think we don't have them on derived pipelines that are made from two or more components (such as Flux + ImageToImage, or AnimateDiff + VideoToVideo)

will address the tests and minor fixes

quick update, @a-r-r-o-w and @yiyixuxu I found a small bug in my inpaint controlnet pipeline for which the tests are failing will address this by today

[a-r-r-o-w]

Just two more change requests

[ighoshsubho]

@yiyixuxu and @a-r-r-o-w fixed the minor bugs and tests should pass now

[mrtpk]

Can't wait for this awesomeness!!!!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[yiyixuxu]

thank you!

[mrtpk]

Thank you @ighoshsubho

[DoiryCool]

How can I use other controlnets, such as XLabs-AI/flux-controlnet-canny-v3? The canny model in the example exceeds my vmem