Batch Size Optimization

[NullSenseStudio]

I've seen great speed improvement when generating multiple images in batches. So far I've tested batch sizes of 2-5 and reduced run time by 26-39% using a GTX 1070 along with the default half precision and attention slicing optimizations.

It's implemented as a new speed optimization option that'll only go into effect if iterations or file batch are used. It'll split up the requested images into as many full batches that it can and do a smaller batch on any remaining at the end.

Progress

• prompt_to_image
• image_to_image
• inpaint
• outpaint
• depth_to_image
• tiled step preview
• Stability SDK
• VAE slicing
• upscale (waiting for Automatic Seamless Detection #410 first)

Issues

This feature may only be suitable for CUDA users right now.
https://huggingface.co/docs/diffusers/optimization/mps#known-issues

Generating multiple prompts in a batch crashes or doesn’t work reliably. We believe this is related to the mps backend in PyTorch. This is being resolved, but for now we recommend to iterate instead of batching.

Only other downsides are extra memory usage and results aren't entirely deterministic.

Testing indicates that the deterministic nature isn't affected by the seeds or prompts of other images in the same batch. The same batch size with the same settings for any single or multiple images in it will result in the same image, only the batch size amount is causing the odd behavior. I'm uncertain if this is a bug in diffusers or this implementation yet.

[NullSenseStudio]

I've found that VAE slicing will allow for a much higher batch size, and I've got more data on how much less time is required for certain batch sizes on a GTX 1070.

Denoising is strictly the time used for the denoising loop, whereas total also includes VAE decoding. Time saving might even be a little higher if I'd factor in the full round trip time from requesting images to them being available on the frontend.

All testing was done with VAE slicing enabled. VAE slicing disabled was actually showing worse performance just with a batch size of 4, but I assume it could be better for cards with more VRAM (though a larger batch size will likely see more time saving than a small enough batch size that allows VAE slicing to be disabled).

I've expanded the scope of this PR to include an optimization toggle for VAE slicing and will also attempt to implement batching in upscaling since it requires so many tiles.

[NullSenseStudio]

Seeing similar gains in upscaling but not as sharp of a performance increase as prompt_to_image.

Using the same optimization settings and the default 128 tile size.

The VAE only runs in float32 and seems to be the main reason for me to not be able to achieve as high of a batch size. I can't even bump the tile size up to 192 without running out of VRAM when decoding latents.

[NullSenseStudio]

Just found out in image_to_image.py the line image = init_image or image is caching the init_image variable when first called and continues to use that regardless of what image is used. Didn't realize I'd accidentally fix a bug when removing it.

[NullSenseStudio]

I'm having trouble trying to implement batching for Stability SDK properly. It's able to accept a list of prompts, but I'm only able to get it to generate images based off the first one. I'll just leave it as simple iteration.

[carson-katri]

Overall looks good, and it also worked on mps. The only thing to look out for is huggingface/diffusers#1909, some schedulers don't handle a list of generators properly yet. Could it be adjusted to pass a single generator if the batch size is 1?

[NullSenseStudio]

I believe that's all the necessary changes. I'll also note that upscaling doesn't use a list of generators when batching like other actions since I don't think seed reproducibility is as necessary there.