[Assistants] Use textToImage task for avatar generation #662
Conversation
nsarrazin
added
enhancement
|
Need to put a front-facing checkbox for the user to use this feature or not when creating an avatar, but as a poc it works quite well. for huggingchat in prod we may need to pin a model for this if possible, to reduce generation latency |
|
I added a user control in the assistant creation form. There's also a 30s timeout on the image generation. |
nsarrazin
changed the title
|
I'd like to review it soon. Could you resolve the conflicts in the meantime? |
|
Should be good @mishig25 ! |
|
From UX-wise, if you click "generate avatar", you shouldn't be able to select local img (i.e, click "click to upload", it should be disabled). Otherwise, it is confusing, which image will be used Screen.Recording.2024-01-12.at.5.46.41.PM.mov |
| ENABLE_ASSISTANTS=false #set to true to enable assistants feature | ||
| ASSISTANTS_GENERATE_AVATAR=true #requires an hf token, uses the model description and name to generate an avatar using a text to image model | ||
| TEXT_TO_IMAGE_MODEL="runwayml/stable-diffusion-v1-5" |
There was a problem hiding this comment.
nit: but we can use newer models like https://huggingface.co/latent-consistency/lcm-lora-ssd-1b (assuming that they are hosted on inference API) that produce higher quality images faster
- It is higher quality because, it is based on SD XL
- It produces images faster, because it is smaller model (ssd) & uses fewer sampling (lcm-lora)
There was a problem hiding this comment.
Ah that's fair! I picked sd1.5 over sdxl because I assumed it'd be faster but if we have faster models available by all mean let's change it 😁
I think we should prio speed>image quality for this feature, most of the time this will be seen as only a small thumbnail, so if you have good model recommendations, feel free!
There was a problem hiding this comment.
for example, https://huggingface.co/latent-consistency/lcm-lora-sdv1-5 would be faster than runway/sd-1.5. https://huggingface.co/latent-consistency/lcm-lora-sdxl might still be faster than runway/sd-1.5. Maybe @patil-suraj @sayakpaul can confirm?
There was a problem hiding this comment.
There was a problem hiding this comment.
Since it's SDXL, it will not be SD speed. You can try out Segmind's SSD-1B.
There was a problem hiding this comment.
So https://huggingface.co/latent-consistency/lcm-lora-ssd-1b would be the best, in terms of speed/quality tradeoff?
There was a problem hiding this comment.
I would say so. But, would also consider playing SD Turbo.
There was a problem hiding this comment.
What's the license of SD Turbo ?
There was a problem hiding this comment.
I tried latent-consistency/lcm-lora-ssd-1bz but it doesn't seem to load, I think it would need to be pinned in the API on our side if we go for it, just a note not necessarily a blocker 😁
Co-authored-by: Mishig <[email protected]>
The intent was that if an image is uploaded, it takes precedence over the checkbox (the checkbox disappears and the image preview is shown instead) but if you think it's clearer to also disable the upload button when the checkbox is checked then I can add that for sure. Do you think it's better if the button is disabled or completely hidden ? |
Co-authored-by: Mishig <[email protected]>
disabled |
done! |
|
There's a feature flag in front of it so I think we can merge this back into the assistants PR and then enable the flag or not in the prod config if we decide to go for it |
* First push on assistants * push fixes * fix add assistant * Sign up works * lint * mobile layout fixes * design fixes * Merge branch 'main' into feature/assistants * fix copy button * add error feedback * hide duplicate feature * remove wrong comments * add autoredirect if assistant is missing * latest changes: - add edit feature - hash assistant avatar - get rid of ugly line - check for non existent avatar - make a better looking upload icon * Update src/routes/conversation/+server.ts Co-authored-by: Mishig <[email protected]> * reused type more cleanly * fix type in shared conversation * fixed feature * fix: share conv with an assistant * delete assistant avatars in db when deleting avatar * affordance on avatar upload * improve assistant conv start on mobile * settings modal fly in * better mobile intro * mobile padding * link affordance * Make assistants disabled by default, but enabled in huggingchat * lint * Fix bottom model name * ui tweaks * Initial work on chat thumbnails * fix build * Get rid of deps * Update src/routes/settings/assistants/[assistantId]/avatar/+server.ts Co-authored-by: Mishig <[email protected]> * add comment to app_base * Use event modifiers * Use CSS uppercase instead everywhere * Update src/lib/components/NavMenu.svelte Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Clearer error message for avatar size check * one less op on flag check * revert back preventDefault change in LoginModal * Update src/routes/settings/+layout.svelte Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Added app logo in corner of thumbnail and clamped description length * improved thumbnails * Remove warnings * Reuse Assisntants settings component (#678) * Update Assisntants settings * format * [Assistants] Use textToImage task for avatar generation (#662) * Generate assistants avatar using stablediffusion * wording * Update +page.server.ts Co-authored-by: Michael Fried <[email protected]> * Add timeout & controls to avatar generation * Add controls for avatar generation in .env * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Update src/lib/components/AssistantSettings.svelte Co-authored-by: Mishig <[email protected]> * Fix avatar gen feature flag * Can only upload avatar if generate is unchecked --------- Co-authored-by: Michael Fried <[email protected]> Co-authored-by: Mishig <[email protected]> * layout * small fixes * hint * Show feature if login is not required * lint * Only show creator name if it's defined * tweaks * thumbnail update * thumbnail font-size * Always display model at the bottom * Bottom links now go to settings * fix lint * silent release * fix bg on share link * [Assistant] Delete avatar button instead of reset (#725) * Add rate-limited image generating endpoint * Add generate avatar button * add little padding for firefox focus ring * format * fix upload image bug * Fix uploads, replace reset by delete * left-align buttons * rm avatar generation feature * final changes to delete feature * sys prompt min height * padding * Add object-cover everywhere --------- Co-authored-by: Victor Mustar <[email protected]> --------- Co-authored-by: Mishig <[email protected]> Co-authored-by: Victor Mustar <[email protected]> Co-authored-by: Michael Fried <[email protected]>
* First push on assistants * push fixes * fix add assistant * Sign up works * lint * mobile layout fixes * design fixes * Merge branch 'main' into feature/assistants * fix copy button * add error feedback * hide duplicate feature * remove wrong comments * add autoredirect if assistant is missing * latest changes: - add edit feature - hash assistant avatar - get rid of ugly line - check for non existent avatar - make a better looking upload icon * Update src/routes/conversation/+server.ts Co-authored-by: Mishig <[email protected]> * reused type more cleanly * fix type in shared conversation * fixed feature * fix: share conv with an assistant * delete assistant avatars in db when deleting avatar * affordance on avatar upload * improve assistant conv start on mobile * settings modal fly in * better mobile intro * mobile padding * link affordance * Make assistants disabled by default, but enabled in huggingchat * lint * Fix bottom model name * ui tweaks * Initial work on chat thumbnails * fix build * Get rid of deps * Update src/routes/settings/assistants/[assistantId]/avatar/+server.ts Co-authored-by: Mishig <[email protected]> * add comment to app_base * Use event modifiers * Use CSS uppercase instead everywhere * Update src/lib/components/NavMenu.svelte Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Clearer error message for avatar size check * one less op on flag check * revert back preventDefault change in LoginModal * Update src/routes/settings/+layout.svelte Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Added app logo in corner of thumbnail and clamped description length * improved thumbnails * Remove warnings * Reuse Assisntants settings component (huggingface#678) * Update Assisntants settings * format * [Assistants] Use textToImage task for avatar generation (huggingface#662) * Generate assistants avatar using stablediffusion * wording * Update +page.server.ts Co-authored-by: Michael Fried <[email protected]> * Add timeout & controls to avatar generation * Add controls for avatar generation in .env * Update src/routes/+layout.server.ts Co-authored-by: Mishig <[email protected]> * Update src/lib/components/AssistantSettings.svelte Co-authored-by: Mishig <[email protected]> * Fix avatar gen feature flag * Can only upload avatar if generate is unchecked --------- Co-authored-by: Michael Fried <[email protected]> Co-authored-by: Mishig <[email protected]> * layout * small fixes * hint * Show feature if login is not required * lint * Only show creator name if it's defined * tweaks * thumbnail update * thumbnail font-size * Always display model at the bottom * Bottom links now go to settings * fix lint * silent release * fix bg on share link * [Assistant] Delete avatar button instead of reset (huggingface#725) * Add rate-limited image generating endpoint * Add generate avatar button * add little padding for firefox focus ring * format * fix upload image bug * Fix uploads, replace reset by delete * left-align buttons * rm avatar generation feature * final changes to delete feature * sys prompt min height * padding * Add object-cover everywhere --------- Co-authored-by: Victor Mustar <[email protected]> --------- Co-authored-by: Mishig <[email protected]> Co-authored-by: Victor Mustar <[email protected]> Co-authored-by: Michael Fried <[email protected]>
This PR adds a togglable feature for generating the assistants avatars using a customizable text to image model.