Base of modular backend #6606

StAlKeR7779 · 2024-07-12T21:53:27Z

Summary

Base code of new modular backend from #6577.
Contains normal generation and regional prompts support.
Also preview extension included to test if extensions logic works.

Related Issues / Discussions

https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d

QA Instructions

Run with and without set USE_MODULAR_DENOISE environment.
Currently only normal and regional conditionings supported, so just generate some images and compare with main output.

Merge Plan

Discuss a bit more about injection point names?
As if for example in future unet will be overridable, current pre_unet/post_unet assumes to name override as unet what feels a bit odd.
Also apply_cfg - future implementation could ignore/not use cfg, so in this case combine_noise_predictions/combine_noise seems more suitable.

Checklist

The PR has a short but descriptive title, suitable for a changelog
Tests added / updated (if applicable)
Documentation added / updated (if applicable)

…ction points

RyanJDick

Nice work 💪

I did an initial round of review. I haven't looked at diffusion_backend.py yet - I just focused on the new backend 'infrastructure'.

I did a really quick smoke test. SD1.5 worked, SDXL failed. I'll do much more testing once this is closer to being merged.

invokeai/app/invocations/denoise_latents.py

invokeai/backend/stable_diffusion/denoise_context.py

invokeai/backend/stable_diffusion/extensions/__init__.py

invokeai/backend/stable_diffusion/diffusers_pipeline.py

invokeai/backend/stable_diffusion/extensions_manager.py

RyanJDick

Another round of review - I got to the remaining code that I hadn't looked at yet.

invokeai/backend/model_patcher.py

invokeai/backend/stable_diffusion/denoise_context.py

invokeai/backend/stable_diffusion/extensions_manager.py

invokeai/backend/stable_diffusion/diffusion/conditioning_data.py

invokeai/app/invocations/denoise_latents.py

invokeai/backend/stable_diffusion/diffusion_backend.py

hipsterusername · 2024-07-18T00:16:54Z

I tested and wasn’t able to replicate getting discrepancies on both sd1.5 and sdxl (I.e., all the same)

RyanJDick · 2024-07-18T13:45:51Z

I dug into the image discrepancies some more. Notes from my tests:

These tests were all conducted without xformers.
Even on main, I sometimes see discrepancies across app restarts. I.e. generating with the same args before and after app restart (to clear the cache) sometimes generates an identical image and sometimes generates a slightly different image.
- I tried setting torch.backends.cudnn.deterministic = True (as recommended here) in the app initialization code, but this did not seem to help.

I wish we understood the source of the non-determinism better, but given that it's present on main as well, I have no reason to believe that this PR introduces any new reproducibility issues.

Co-Authored-By: Ryan Dick <[email protected]>

## Summary Fix function call that we forgot to update in #6606 ## QA Instructions Run a TiledMultiDiffusionDenoiseLatents invocation and make sure it doesn't crash. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [x] _Tests added / updated (if applicable)_ - [x] _Documentation added / updated (if applicable)_

## Summary Rescale CFG code from #6577. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ~~Note: for some reasons there slightly different output from run to run, but I able sometimes to get same output on main and this branch.~~ Fix presented in #6641. ## Merge Plan ~~Nope.~~ Merge #6641 firstly, to be able see output difference properly. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary FreeU code from #6577. Also fix issue with sometimes slightly different output. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ## Merge Plan Nope. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary ControlNet code from #6577. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ## Merge Plan Merge #6641 firstly, to be able see output difference properly. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary Seamless code from #6577. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ## Merge Plan Nope. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary T2I Adapter code from #6577. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ## Merge Plan Nope. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary Code for inpainting and inpaint models handling from #6577. Separated in 2 extensions as discussed briefly before, so wait for discussion about such implementation. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. Try and compare outputs between backends in cases: - Normal generation on inpaint model - Inpainting on inpaint model - Inpainting on normal model ## Merge Plan Nope. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

## Summary Code for lora patching from #6577. Additionally made it the way, that lora can patch not only `weight`, but also `bias`, because saw some loras which doing it. ## Related Issues / Discussions #6606 https://invokeai.notion.site/Modular-Stable-Diffusion-Backend-Design-Document-e8952daab5d5472faecdc4a72d377b0d ## QA Instructions Run with and without set `USE_MODULAR_DENOISE` environment. ## Merge Plan Replace old lora patcher with new after review done. If you think that there should be some kind of tests - feel free to add. ## Checklist - [x] _The PR has a short but descriptive title, suitable for a changelog_ - [ ] _Tests added / updated (if applicable)_ - [ ] _Documentation added / updated (if applicable)_

StAlKeR7779 added 8 commits July 12, 2024 20:31

Base code from draft PR

9cc852c

A bit rework conditioning convert to unet kwargs

0bc6037

Rename modifiers to callbacks, convert order to int, a bit unify inje…

87e96e1

…ction points

Simplify guidance modes

bd8ae5d

Renames

3a9dda9

Remove overrides logic for now

7e00526

Remove remains of priority logic

e961dd1

Add preview extension to check logic

499e4d4

github-actions bot added python PRs that change python files invocations PRs that change invocations backend PRs that change backend files labels Jul 12, 2024

StAlKeR7779 marked this pull request as ready for review July 12, 2024 22:55

StAlKeR7779 requested review from lstein, blessedcoolant, brandonrising, RyanJDick, hipsterusername and psychedelicious as code owners July 12, 2024 22:55

RyanJDick requested changes Jul 15, 2024

View reviewed changes

StAlKeR7779 added 6 commits July 16, 2024 00:31

Fix condtionings logic

d623bd4

Remove 'del' operator overload

fd8d1c1

Multiple small fixes

9f088d1

Separate inputs in denoise context

608cbe3

Change attention processor apply logic

cec345c

Added some comments

b7c6c63

RyanJDick reviewed Jul 16, 2024

View reviewed changes

StAlKeR7779 added 4 commits July 17, 2024 03:24

Rename sequential as private variable

cd1bc15

Move out _concat_conditionings_for_batch submethods

ae6d4fb

Convert conditioning_mode to enum

03e22c2

Remove patch_unet logic for now

137202b

StAlKeR7779 and others added 2 commits July 18, 2024 23:49

Ryan's suggested changes to extension manager/extensions

0c56d4a

Co-Authored-By: Ryan Dick <[email protected]>

Add unit tests for ExtensionsManager and ExtensionBase.

83a86ab

github-actions bot added the python-tests PRs that change python tests label Jul 19, 2024

StAlKeR7779 and others added 2 commits July 19, 2024 23:17

Add invocation cancellation logic to patchers

39e10d8

Merge branch 'main' into stalker-backend_base

78d2b1b

RyanJDick approved these changes Jul 19, 2024

View reviewed changes

hipsterusername approved these changes Jul 19, 2024

View reviewed changes

RyanJDick enabled auto-merge July 19, 2024 20:32

RyanJDick merged commit 473f4cc into invoke-ai:main Jul 19, 2024
14 checks passed

RyanJDick added a commit that referenced this pull request Jul 19, 2024

Fix function call that we forgot to update in #6606

a8cc5ca

RyanJDick mentioned this pull request Jul 19, 2024

Fix function call that we forgot to update in #6606 #6636

Merged

3 tasks

This was referenced Jul 21, 2024

Modular backend - add rescale cfg #6640

Merged

Modular backend - add FreeU #6641

Merged

Modular backend - add ControlNet #6642

Merged

Modular backend - inpaint #6643

Merged

chainchompa pushed a commit that referenced this pull request Jul 22, 2024

Fix function call that we forgot to update in #6606

bb13204

This was referenced Jul 23, 2024

Modular backend - Seamless #6651

Merged

Modular backend - T2I Adapter #6662

Merged

Modular backend - LoRA/LyCORIS #6667

Merged

Modular backend - overrides #6692

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Base of modular backend #6606

Base of modular backend #6606

StAlKeR7779 commented Jul 12, 2024 •

edited by RyanJDick

Loading

RyanJDick left a comment

RyanJDick left a comment

hipsterusername commented Jul 18, 2024 •

edited

Loading

RyanJDick commented Jul 18, 2024

Base of modular backend #6606

Base of modular backend #6606

Conversation

StAlKeR7779 commented Jul 12, 2024 • edited by RyanJDick Loading

Summary

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

RyanJDick left a comment

Choose a reason for hiding this comment

RyanJDick left a comment

Choose a reason for hiding this comment

hipsterusername commented Jul 18, 2024 • edited Loading

RyanJDick commented Jul 18, 2024

StAlKeR7779 commented Jul 12, 2024 •

edited by RyanJDick

Loading

hipsterusername commented Jul 18, 2024 •

edited

Loading