Feature - Cross Attention Control for Stable Diffusion

[FahimF]

Short Description
Any possibility of getting Cross Attention Control support added for Stable Diffusion as seen implemented as per the repo linked below on the PyTorch side?
https://github.com/bloc97/CrossAttentionControl

I'd be happy with just the ability to use a prompt edit and to be able to modify the output from the original prompt. It makes a huge difference in the types of images you can generate and how you can guide the final image to be what you want ...

Papers
https://arxiv.org/abs/2208.01626

Existing Implementations
https://github.com/bloc97/CrossAttentionControl

Other Information
I've looked at trying to implement it myself but I'm afraid that my PyTorch and Keras/Tensorflow knowledge isn't sufficiently advanced enough to know where to start 😄 But if you think this would be a good feature, and somebody wants to give me a few pointers on where to start, I can give it a shot.

[FahimF]

I might have managed to figure out some of the logic for the Keras side, but I don't think my implementation is totally correct. Here's where my code is at, at the moment. If anybody has any pointers on what I might be doing wrong, would appreciate it 😄

All the changes are in the StableDiffusion class - I added a new prompt_edit parameter and added the logic for incorporating the prompt_edit when there is actually a prompt_edit parameter. It kind of works, but my suspicion is that I got the final inference logic wrong and so the inference is just for the prompt_edit rather than using Cross Attention Control as is the intention ...

stable_diffusion.txt

[tanzhenyu]

From our roadmap, Q4 we have been focusing on image impainting and video generation. We can open this up for external contribution

[sachinprasadhs]

Thanks for reporting the issue! We have consolidated the development of KerasCV into the new KerasHub package, which supports image, text, and multi-modal models. Please read keras-team/keras-hub#1831. KerasHub will support all the core functionality of KerasCV.

KerasHub can be installed with !pip install -U keras-hub. Documentation and guides are available at keras.io/keras_hub.

With our focus shifted to KerasHub, we are not planning any further development or releases in KerasCV. If you encounter a KerasCV feature that is missing from KerasHub, or would like to propose an addition to the library, please file an issue with KerasHub.

[github-actions]

This issue is stale because it has been open for 14 days with no activity. It will be closed if no further activity occurs. Thank you.