[Community Pipeline] Speech to Image

[osanseviero]

Intro

Community Pipelines are introduced in diffusers==0.4.0 with the idea of allowing the community to quickly add, integrate, and share their custom pipelines on top of diffusers.

You can find a guide about Community Pipelines here. You can also find all the community examples under examples/community/. If you have questions about the Community Pipelines feature, please head to the parent issue.

Idea: Speech to Image

You can use a transformer automatic-speech-recognition such as OpenAI whisper to transcribe the text, and pass that to Stable Diffusion. Together, this would create a nice speech-to-image pipeline.

Resources

• https://twitter.com/art_zucker/status/1579881525071728642
• https://huggingface.co/docs/transformers/model_doc/whisper

[MikailINTech]

Hi, would be glad to work on it

[osanseviero]

Awesome! Feel free to open a PR and if you have any questions just let us know :) (ideally in the parent issue unless it's very specific to this 🙏 )

[akashAD98]

I’m very much interested for this,I’ll try this feature

[MikailINTech]

Hi @osanseviero, I opened a PR for this issue. For the moment, I have tested my pipeline for whisper and it works pretty well. I'm waiting for your suggestions (maybe making the pipeline more generic so that we can use other speech-recognition models ?)

[patrickvonplaten]

It works very well ! Well done @MikailINTech :-)

[fffiloni]

📌 gradio demo is available here: https://huggingface.co/spaces/fffiloni/speech-to-image

[osanseviero]

This is amazing @fffiloni! Great work!

[osanseviero]

Closing this issue as this is now done 🔥 amazing work everyone!

[darwinharianto]

I tried the sample from https://github.com/huggingface/diffusers/tree/main/examples/community#speech-to-image
it gives me,

Is there anything wrong with the code?