create faces with stylegan and dalle?

[molo32]

amazing job, I was wondering if anyone has a colab notebook to create faces from text with or without stylegan2 by ex.

[lucidrains]

@molo32 yes! it is now possible, but DALLE is not needed. by simply combining CLIP with Stylegan2, we can now summon images from the latent space of a trained generator. I will add it to my stylegan2 repository https://github.com/lucidrains/stylegan2-pytorch in due time :) focused on equivariant attention for alphafold2 at the moment

[lucidrains]

@molo32 somebody else already did it :) you can try it at https://twitter.com/advadnoun/status/1353453719510163459?s=20

[powderblock]

@molo32 somebody else already did it :) you can try it at https://twitter.com/advadnoun/status/1353453719510163459?s=20

wow this is awesome!!! any idea how to use this for faces?

[lucidrains]

@powderblock https://twitter.com/pbaylies/status/1348313176115458048?s=20

[lucidrains]

@powderblock it'll work with any generator! the latent space has suddenly become infinitely more traversable, by way of another neural network as the guide :)

[rom1504]

Is this done simply by generating a batch of images with a generator, ranking them with clip and trying again randomly until the dot product is high enough ?
Or could this be done instead by retropropagating the error like stylegan encoder is doing ? (https://github.com/Puzer/stylegan-encoder)

[lucidrains]

@rom1504 both!

[rom1504]

Using a multi modal encoder trained for similarity and a generator in that way seems like a really powerful idea.
I wonder if clip would work to generate very accurate descriptions too (take a picture, use a language model to generate text, retropropagate... Until you get good enough dot product with the picture)
And more generally, if we have more such multi modal encoders (text audio, text 3d model, ...), it seems to open the gate to generating almost anything.

[lucidrains]

@rom1504 if you follow Mario @ quasimodo on twitter, he was able to coax out text from CLIP. i think the surer thing to do is to rank text generation from existing caption to text transformers, as an alternative to beam search

[lucidrains]

@rom1504 yes, multimodal is here, attention was all we need

[rom1504]

Is that https://twitter.com/Quasimodo ? Seems to be unavailable

[lucidrains]

oops, @ quasimondo