[Feature Request]: Latent diffusion upscaler for the Stable Diffusion autoencoder

[specblades]

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

Can we implement it?

Q from Twitter @RiversHaveWings:
I've trained a latent diffusion upscaler for the Stable Diffusion autoencoder (and anything you feel like feeding into it if you can tolerate a little artifacts) in collaboration with
@stabilityai
. Try the Colab written by
@nshepperd1
https://colab.research.google.com/drive/1o1qYJcFeywzCIdkfKJy7cTpgZTCM2EI4

Proposed workflow

See in colab

Additional information

No response

[Ehplodor]

Is it similar to the "scale latents" option for A1111's highres fix ? (see #2613 for example for some info)

[specblades]

Is it similar to the "scale latents" option for A1111's highres fix ? (see #2613 for example for some info)

Nope

[pbaylies]

I tested this out, wasn't hard to convert it to a stand-alone script. Note that there is a stray config file used from here - https://huggingface.co/spaces/multimodalart/latentdiffusion/blob/main/latent-diffusion/models/first_stage_models/kl-f8/config.yaml (not the CompVis repo), but just putting that file there did work.

Also added .half() in a bunch of places to use less VRAM, etc.
sdu.txt

[specblades]

I tested this out, wasn't hard to convert it to a stand-alone script. Note that there is a stray config file used from here - https://huggingface.co/spaces/multimodalart/latentdiffusion/blob/main/latent-diffusion/models/first_stage_models/kl-f8/config.yaml (not the CompVis repo), but just putting that file there did work.

Also added .half() in a bunch of places to use less VRAM, etc. sdu.txt

Can u explane for newbie how to use it?
I cant get through

[pbaylies]

@specblades probably better to start with the notebook, then, it will be friendlier to use.

[specblades]

@specblades probably better to start with the notebook, then, it will be friendlier to use.

Goal is to use it in webui, i mean
nvm, mb somebody will add it

[specblades]

Little comparison x2 upscale: native/ldsr (50 steps)/autoencoder upscale (48 steps)

The last is much sharper, but with artefacts. I like it more.
And its faster by ~10-20x
We rly need it in a1111!

[specblades]

Emad retweet it.
Hero, pls, make it work in a1111

[nagolinc]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files:
https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json
https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

[specblades]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files: https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

Could you provide some kind of tutorial on how to use it?
There are no ui for change upscale ammount or steps

[specblades]

Okay, I was able to convert @pbaylies sdu.txt into a script for automatic1111

https://gist.github.com/nagolinc/3993e7329cafab5d5bd4698977ebebcc

Before you can run it, you will need to download the two files: https://models.rivershavewings.workers.dev/config_laion_text_cond_latent_upscaler_2.json https://models.rivershavewings.workers.dev/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth

into your {automatic}/models/LDSR/ folder

Many OOMs, "RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same" and other errors. Does not save the result automatically.

[arpitest]

Exception: bad file inside ./models/LDSR/laion_text_cond_latent_upscaler_2_1_00470000_slim.pth: laion_text_cond_latent_upscaler_2_1_00470000_slim/data.pkl

The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.

[arpitest]

it start with --disable-safe-unpickle, then works only once, then restart required to work again once :( but it's good for testing, i like the result very much!
(the second run gives the "Input type (torch.cuda.HalfTensor) and weight type (torch.HalfTensor) should be the same" error)

could it be implemented as an upscaler, to be usable in Extras tab too, instead of being a script?
if it's a latent diffusion upscaler (as the twitter says), maybe it's enough to add an LDSR model selector to settings, to use this version of model?

[arpitest]

hmm this is also interetsing: "[Stability AI]Nov 10
That means that if you want to use it with Stable Diffusion, you take the generated "latent" and pass into the upscaler before decoding with your standard VAE."

[arpitest]

i've done some cleanup on the script, removed unused imports and functions, and moved a few model.to() around, and now it works fine, no crash, feel free to test:
http://thot.banki.hu/arpi/sdu_upscale.py

also added some test code to main() so it can be used standalone from cli.
next step will be adding it to extras as upscaler :) no idea how to do that yet...

[arpitest]

i've replaced the bilinear interpolate function in processing.py (for highres fix latent scaling) to a call to this new model (which operates in latent space anyway), and IT WORKS! if you enable highres fix & scale latent, and set denoising to zero, then it will immediately upscale by 2x the generated image! if you set denoising to some low value <0.3 then it will work further on the upscaled version and fix some artifacts too!

!!!!!!!!!!!!!!!!!! UPSCALING LATENT !!!!!!!!!!!!!!!!!!!!!
['Portrait photo of Goddess']
before: torch.Size([1, 4, 64, 64])
42it [00:06, 6.75it/s]
after: torch.Size([1, 4, 128, 128])
!!!!!!!!!!!!!!!!!! UPSCALING DONE !!!!!!!!!!!!!!!!!!!!!

@AUTOMATIC1111 please look at run_sdu_latent_upscale() in http://thot.banki.hu/arpi/sdu_upscale_mod.py
it can be used as replacement in processing.py instead of torch.nn.functional.interpolate(samples, ..., mode="bilinear")

[specblades]

i've replaced the bilinear interpolate function in processing.py (for highres fix latent scaling) to a call to this new model (which operates in latent space anyway), and IT WORKS! if you enable highres fix & scale latent, and set denoising to zero, then it will immediately upscale by 2x the generated image! if you set denoising to some low value <0.3 then it will work further on the upscaled version and fix some artifacts too!

!!!!!!!!!!!!!!!!!! UPSCALING LATENT !!!!!!!!!!!!!!!!!!!!! ['Portrait photo of Goddess'] before: torch.Size([1, 4, 64, 64]) 42it [00:06, 6.75it/s] after: torch.Size([1, 4, 128, 128]) !!!!!!!!!!!!!!!!!! UPSCALING DONE !!!!!!!!!!!!!!!!!!!!!

@AUTOMATIC1111 please look at run_sdu_latent_upscale() in http://thot.banki.hu/arpi/sdu_upscale_mod.py it can be used as replacement in processing.py instead of torch.nn.functional.interpolate(samples, ..., mode="bilinear")

You can re-create the discussion so you have the authorship

[arpitest]

hmm, it even works for 4x upscaling (calling the model twice on latents), the 4x version reminds me to old midjourney style... unfortunately 2048x2048 resulted cuda out of memory, so the example here is 512x384, original, 2x and 4x upscaled in latent space:

i really love this upscaler! it has an unique style at 4x but still better than other methods:

[specblades]

@arpitest
i dunno, a1111 wont load it in t2i/i2i

COMMANDLINE_ARGS= --disable-safe-unpickle --xformers --allow-code

[arpitest]

@specblades the _mod version is not a script, it should be in modules/ and the func called from hires-fix part of processing.py, needs some small changes to the code... i hope @AUTOMATIC1111 will do it soon in proper way, instead of my ugly hack :)

[specblades]

@arpitest i understand
can u modify processing.py to load it?
or it will work if i put it in modules folder?

i think we need both - upscale in extras and hr-fix

[arpitest]

@arpitest i understand can u modify processing.py to load it? or it will work if i put it in modules folder?

i think we need both - upscale in extras and hr-fix

here is my modified version: http://thot.banki.hu/arpi/processing.py
copy this (replace original) and the sdu_upscale_mod.py to modules/
then restart, enable scale latent in settings and hires fix at txt2img, set denoising=0 then generate something :)
(it has 4x scaling now, if you want 2x then remove one of the 2 calls to run_sdu_latent_upscale)

sorry this is a proof of concept only, not intended for wide use this ugly way :)

[specblades]

@arpitest
i want to try but

nvm, do ur best, please!
You are awesome!

[arpitest]

even the 4x is looking pretty good for some images:

[Ehplodor]

Out of curiosity, does anybody knows of some comparison between this new latent upscaler and the "scale latent hires-fix" already implemented in A1111 (that performs bilinear interpolation in latent space) ?

[arpitest]

(that performs bilinear interpolation in latent space) ?

using bilinear interpolation on vectors (latents are not pixels!) is a bad idea anyway... there are methods for vector interpolation, like euler, quaternion etc. it's a very different math...

[Ehplodor]

(that performs bilinear interpolation in latent space) ?

using bilinear interpolation on vectors (latents are not pixels!) is a bad idea anyway... there are methods for vector interpolation, like euler, quaternion etc. it's a very different math...

I see a way for improving highres-fix's scale latents option here. TY

[Ehplodor]

(that performs bilinear interpolation in latent space) ?

using bilinear interpolation on vectors (latents are not pixels!) is a bad idea anyway... there are methods for vector interpolation, like euler, quaternion etc. it's a very different math...

I see a way for improving highres-fix's scale latents option here. TY

FYI : feature request for new latent vector interpolation methods here

[arpitest]

formula conditioning=denoising/2

it looks bad... i've found the best formula conditioning=1.0-denoising

for higher denoising value you need lower conditioning to compensate

[aleksusklim]

conditioning=1.0-denoising
for higher denoising value you need lower conditioning to compensate

At high denoisings it will oversaturate like CFG>20
In this regard, you can just keep always conditioning=0.0 and call it a day, since it will totally eliminate duplication artifacts.

[DrMacabre]

broken since last update. I'm getting an error whenever the modified processing.py is used after my last git pull today

Upscale latent space image when doing hires. fix no longer available in webui settings btw

[arpitest]

broken since last update. I'm getting an error whenever the modified processing.py is used after my last git pull today

Upscale latent space image when doing hires. fix no longer available in webui settings btw

which repo/branch? it is still working for me. what error?

[DrMacabre]

which repo/branch? it is still working for me. what error?
`
Automatic1111
SD-WebUI revision: 685f963

File "F:\Programs\stable-diffusion-webui\launch.py", line 295, in
start()
File "F:\Programs\stable-diffusion-webui\launch.py", line 290, in start
webui.webui()
File "F:\Programs\stable-diffusion-webui\webui.py", line 132, in webui
initialize()
File "F:\Programs\stable-diffusion-webui\webui.py", line 52, in initialize
modules.sd_models.setup_model()
File "F:\Programs\stable-diffusion-webui\modules\sd_models.py", line 42, in setup_model
enable_midas_autodownload()
File "F:\Programs\stable-diffusion-webui\modules\sd_models.py", line 251, in enable_midas_autodownload
for k, v in midas.api.ISL_PATHS.items():
AttributeError: module 'ldm.modules.midas' has no attribute 'api'

[arpitest]

are you overwriting the processing.py by the old patched version or patching the new one?
now updated the patched version too, but i recommended using the patch over the actual github file:
http://thot.banki.hu/arpi/processing.diff

[DrMacabre]

are you overwriting the processing.py by the old patched version or patching the new one? now updated the patched version too, but i recommended using the patch over the actual github file: http://thot.banki.hu/arpi/processing.diff

until now i was using the old patched version, how do i use the diff file ?
its working with the new patched file btw, thanks.

[jnpatrick99]

@arpitest i understand can u modify processing.py to load it? or it will work if i put it in modules folder?
i think we need both - upscale in extras and hr-fix

here is my modified version: http://thot.banki.hu/arpi/processing.py copy this (replace original) and the

Thanks for this one! Works really great on square images, but when I try to switch to rectangular like 768x1024 then it crashes at return K.sampling.sample_dpm_adaptive(model_wrap, noise * sigma_max, sigma_min, sigma_max, extra_args=extra_args, eta=eta, **sampler_opts) the error is RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 36 but got size 37 for tensor number 1 in the list.. Is there a way to fix it somehow?

[bbecausereasonss]

Can someone please create simplified instructions on making this work?

[DrMacabre]

How can we still use this mod with latest version of webui?
Right now it gives TypeError: StableDiffusionProcessing.init() got an unexpected keyword argument 'hr_scale'

[Skifoid]

easiest way:

copy to modules/ http://thot.banki.hu/arpi/sdu_upscale_mod.py

add from modules.sdu_upscale_mod import run_sdu_latent_upscale to begining processing.py
and change some code

        if latent_scale_mode is not None:
            for i in range(samples.shape[0]):
                save_intermediate(samples, i)

            samples = torch.nn.functional.interpolate(samples, size=(target_height // opt_f, target_width // opt_f), mode=latent_scale_mode["mode"], antialias=latent_scale_mode["antialias"])

to

        if latent_scale_mode is not None:
            for i in range(samples.shape[0]):
                save_intermediate(samples, i)

            #samples = torch.nn.functional.interpolate(samples, size=(target_height // opt_f, target_width // opt_f), mode=latent_scale_mode["mode"], antialias=latent_scale_mode["antialias"])
            
            print(f"UPSCALING-1 - [{self.height} x {self.width}] [{target_height} x {target_width}]")
            samples=run_sdu_latent_upscale(samples,prompts[0], size=(self.height // opt_f, self.width // opt_f) )

[DrMacabre]

add from modules.sdu_upscale_mod import run_sdu_latent_upscale to begining processing.py
and change some code

it doesn't seem to work, now it gives me this error : RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 42 but got size 43 for tensor number 1 in the list.

EDIT : nevermind, i was using exotic resolution. it's working fine, thank you.

[Danamir]

I'm trying to use the ratio defined in hr fix (1.25 in my tests) instead of 2.0 as latent upscale, but I have no idea on how to proceed. Maybe scaling by 2.0 then downsizing afterward ?

[arpitest]

I'm trying to use the ratio defined in hr fix (1.25 in my tests) instead of 2.0 as latent upscale, but I have no idea on how to proceed. Maybe scaling by 2.0 then downsizing afterward ?

this is fixed 2x scale, cannot do other sizes. and the original size must be divisable by 64 (so the resulting size by 128).

[DrMacabre]

this is fixed 2x scale, cannot do other sizes

are you sure? cause i just tried x1.25 and x2.5 both from 768x896 source size.

[Danamir]

Oh. Too bad, 2.0 is really too much to be useful as hr fix in my opinion. Either you have a very small source size and it contains less details, or you have a too large destination size and the model may be generating incoherent pictures.

[DrMacabre]

Oh. Too bad, 2.0 is really too much to be useful as hr fix in my opinion. Either you have a very small source size and it contains less details, or you have a too large destination size and the model may be generating incoherent pictures.

please check post above yours.

[Danamir]

Oh. Too bad, 2.0 is really too much to be useful as hr fix in my opinion. Either you have a very small source size and it contains less details, or you have a too large destination size and the model may be generating incoherent pictures.

please check post above yours.

Damn, it's really not working on my system. I just tried with your source resolution and 1.25 ratio, here are the outputs :

Are you using the latest Auto1111 commit ? I don't know what I'm doing wrong.

[DrMacabre]

Oh. Too bad, 2.0 is really too much to be useful as hr fix in my opinion. Either you have a very small source size and it contains less details, or you have a too large destination size and the model may be generating incoherent pictures.

please check post above yours.

Damn, it's really not working on my system. I just tried with your source resolution and 1.25 ratio, here are the outputs :

Are you using the latest Auto1111 commit ? I don't know what I'm doing wrong.

yes, latest A1111, i have no idea why it works for me, are you using anything else than 0 on the resize sliders ?

[Danamir]

Yes I have the exact same settings, and the code from http://thot.banki.hu/arpi/sdu_upscale_mod.py .

Very strange. Anyone else having any success with 1.25 ratio ?

[Danamir]

I tried everything I could think of, and it always give me a x2.0 scale.
Could you paste the content of your processing.py, sdu_upscale_mod.py, and the log of an upscale please ?

My upload log for a 768x896 picture x1.25 is :

UPSCALING-1 - [896 x 768] [1120 x 960]████████████████                                 | 16/32 [00:09<00:08,  1.80it/s]
torch.Size([1, 4, 112, 96]) (112, 96)
54it [00:16,  3.33it/s]
torch.Size([1, 4, 224, 192])

The first torch.Size log is (112, 96) which correspond to 869x768 (multiply by 8). But the second log of torch.Size is always doubled ; whatever my original hr fix ratio was.

[arpitest]

the code in sdu_upscale_mod.py always do 2x upscale, it's fixed by design. it does not even know your scaling settings, only gets the original size and the latent matrix:

[batch_size, C, H, W] = low_res_latent.shape
...
x_shape = [batch_size, C, 2H, 2W]

if you get different scale, then it is not in use!

[Danamir]

the code in sdu_upscale_mod.py always do 2x upscale, it's fixed by design. it does not even know your scaling settings, only gets the original size and the latent matrix:

[batch_size, C, H, W] = low_res_latent.shape ... x_shape = [batch_size, C, 2_H, 2_W]

if you get different scale, then it is not in use!

Next thing I tried was to downscale the samples after the 2x scale by using the old interpolate call :

samples=run_sdu_latent_upscale(samples,prompts[0], size=(self.height // opt_f, self.width // opt_f) )
samples = torch.nn.functional.interpolate(samples, size=(target_height // opt_f, target_width // opt_f), mode=latent_scale_mode["mode"], antialias=latent_scale_mode["antialias"])

It keeps the correct target resolution, but it only does a marginally better job at lower denoising (<0.5). And it takes quite a while to compute.

Isn't there any way to do a better downscale before the image generation ?

[AugmentedRealityCat]

Yes I have the exact same settings, and the code from http://thot.banki.hu/arpi/sdu_upscale_mod.py .

Very strange. Anyone else having any success with 1.25 ratio ?

No success for me either - no matter what value I enter as a ratio parameter, the result is always 2.0.

[nekoworkshop]

I'm probably stating something obvious but just a heads up that this new upscaler is not a silver bullet. In my use case it consistently produces images with less details compared to the Latent (nearest) upscaler, especially with DPM++ 2M sampler.

I do believe it can be useful in some workflow or styles, not to mention right now the prototype still has room of turning. Definitely going to keep an eye on it.

[muerrilla]

Hey! Any of you guys still using this? It was my favorite upscaling method but I'm getting an error on the newer webui versions,

EDIT: It has to do with changes in the k-diffusion repo, I guess.

    Traceback (most recent call last):
      File "E:\A1111\stable-diffusion-webui\modules\call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
      File "E:\A1111\stable-diffusion-webui\modules\call_queue.py", line 36, in f
        res = func(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
      File "E:\A1111\stable-diffusion-webui\modules\processing.py", line 739, in process_images
        res = process_images_inner(p)
      File "E:\A1111\stable-diffusion-webui\modules\processing.py", line 896, in process_images_inner
        samples_ddim=sdu_upscale_mod.run_sdu_latent_upscale(samples_ddim)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\modules\sdu_upscale_mod.py", line 313, in run_sdu_latent_upscale
        up_latents = do_sample(noise, extra_args)
      File "E:\A1111\stable-diffusion-webui\modules\sdu_upscale_mod.py", line 297, in do_sample
        return K.sampling.sample_euler_ancestral(model_wrap, noise * sigma_max, sigmas, extra_args=extra_args, eta=eta)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
        denoised = model(x, sigmas[i] * s_in, **extra_args)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\modules\sdu_upscale_mod.py", line 103, in forward
        return self.inner_model(x, sigma, low_res=low_res, low_res_sigma=low_res_sigma, c=c_in)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\layers.py", line 47, in forward
        return self.inner_model(input * c_in, sigma, **kwargs) * c_out + input * c_skip
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\modules\sdu_upscale_mod.py", line 42, in forward
        return self.inner_model(input.half(), sigma.half(), unet_cond=low_res_in.half(), mapping_cond=mapping_cond.half(), cross_cond=cross_cond.half(), cross_cond_padding=cross_cond_padding.half(), **kwargs)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\models\image_v1.py", line 129, in forward
        input = self.u_net(input, cond)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\layers.py", line 258, in forward
        input = block(input, cond)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\layers.py", line 102, in forward
        input = module(input, cond)
      File "E:\A1111\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
      File "E:\A1111\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\layers.py", line 150, in forward
        y = F.scaled_dot_product_attention(q, k, v, dropout_p=self.dropout.p)
    RuntimeError: query: last dimension must be contiguous