Llama support

[SharanSMenon]

So, will we see support for other models, like llama? Maybe through llama.cpp or langchain, or Ollama. Cause continue.dev has support for Ollama and such so I was wondering if models like Llama 2 is on the roadmap?

[wsxiaoys]

We won't integrate any models above 4b for completion at the moment because of the tradeoff between latency, serving cost, and quality.

For chat and Q&A purposes, models like llama are more suitable as they have lower latency requirements. This is being tracked in issue #222

[06kellyjac]

I guess this would also mean code llama at 7b minimum wouldn't fly either

[wsxiaoys]

In issue #370, we introduced support for the Code Llama feature. However, its performance is currently suboptimal due to the significantly larger model size.

[wsxiaoys]

related: ggerganov/llama.cpp#2768 (comment)

Performance on M1 Metal shader seems superior with ggml

[wsxiaoys]

llama has been fully supported since v0.1.0