Model wishlist

[nsarrazin]

Hey everyone!

Just opening an issue to track which models people would like to see supported with Serge.

• Alpaca 7B, 13B & 30B
• Alpaca 7B-Native
• gpt4all
• Vicuna 7B & 13B
• Open Assistant 13B & 30B

Are there any others you would like to see ?

[kolabearafk]

How about WizardLM (https://github.com/nlpxucan/WizardLM) ? Can this new model be supported in Serge please? I heard it's very fast and generates good results.

[CrazyBonze]

I was going to say WizardLM, its been getting some good reviews.

[OrcVole]

https://huggingface.co/reeducator/vicuna-13b-free/discussions

It tries to reduce censorship

[fishscene]

Google PaLM looks interesting.
https://9to5google.com/2023/05/10/google-palm-2/

However, as far as I can tell, the framework is open source and might be available? But I can’t find any info on the training data. I’m also very new to AI and I might be looking in the wrong places for the data needed to be usable for Serge.

[rendel]

MPT-7B: https://github.com/mosaicml/llm-foundry

[noproto]

I'd really like to see support for Guanaco models.
Edit: PR #334

[ethdig]

https://huggingface.co/medalpaca
https://huggingface.co/medalpaca/medalpaca-13b/tree/main
https://huggingface.co/ehartford/Wizard-Vicuna-30B-Uncensored/tree/main

Thank you

[amiravni]

https://huggingface.co/timdettmers/guanaco-33b-merged
https://huggingface.co/tiiuae/falcon-40b

Thanks!

[Betanu701]

from my testing, it seems like all the Q4_0 and the Q8_0 work the best/fastest. The K varieties take too long with CPU only.
Honestly I think it is a toss up between Q4 and Q8. Both seem to run about the same within a margin of error.

[wuast94]

https://huggingface.co/philschmid/instruct-igel-001

[specked]

CodeGen2.5

[kagrith]

from my testing, it seems like all the Q4_0 and the Q8_0 work the best/fastest. The K varieties take too long with CPU only. Honestly I think it is a toss up between Q4 and Q8. Both seem to run about the same within a margin of error.

I get this odd error when trying to run Wizard.

llama.cpp: loading model from /usr/src/app/weights/Wizard-30B-Q4_1.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model

Did you get this?
How did you resolve?

[tonyhardcode]

WizardCoder would be nice to have!

https://huggingface.co/WizardLM/WizardCoder-15B-V1.0

[laurentgoncalves]

Very interested to have the just released code llama : https://ai.meta.com/blog/code-llama-large-language-model-coding/

[gaby]

Fixed via #866