roadmap: Jan supports Deepseek Distilled Models for v0.5.14

[imtuyethan]

Goal

• Jan supports distilled models
• Out-of-scope: Deepseek R1 (wait for llama.cpp support)

Problems

• Problem 1: We are unable to pick up the chat_template from GGUF file due to Cortex's design
  • GGUF uses a Jinja2 template
  • We currently parse Jinja templates into Cortex's format
  • This causes a lot of compatibility issues
• Problem 2: We do not persist the prompt_template changes in Jan
  • User has to change it every time

Workaround

We will do a quick fix in the short-term, while working to align Cortex in the long-term

• Solution 1: We quantize the model ourselves with our own prompt_template
  • janhq/deepseek-distill-qwen2
  • main branch can hold the legacy GGUF format (to support current Jan)
  • Include it in Jan's Hub
• Solution 2: User has to paste in a system prompt template into Jan (which is not 100% correct)
  • bartowski/...

Tasklist

• Models

[olimart]

hi, thanks your work.
do you mind sharing how to set up Deepseek R1 through Llama.cpp ?
It seems that Llama.cpp supports Deepseek models (at least could not find any blocker) but not sure how to get started.

I tried to import straight from the hub but looks like Deepseek R1 is not a GGUF model. Thanks