'prompt_template' not work when using openai/completion

[wapleeeeee]

#3305

When use 'openai/completion' for FIM, found that prompt_template not work.

My ~/.tabby/config.toml is

# Completion model
[model.completion.http]
kind = "openai/completion"
model_name = "minicpm3-4b"
api_endpoint = "http://localhost:8015/v1"
api_key = "xxxxx"
prompt_template = "<|fim_prefix|> {{prefix}} <|fim_suffix|> {{suffix}} <|fim_middle|>"

# Chat model
[model.chat.http]
xxx

But the server got

b'{"model":"minicpm3-4b","prompt":"def fib(n):\\n    if","suffix":"\\n\\n    return fib(n - 1) + fib(n - 2)","max_tokens":64,"temperature":0.1,"stream":true,"presence_penalty":0.0}'

when I use IDE code completion.

It seems that prompt_template not work, since the prompt only contain prefix. Meanwhile the "suffix" param cause 400 Bad request.

I have no idea how to fix that.

[wapleeeeee]

I've tried sglang for the replacement of openai server, but it does not work either.

it gets an error before sending request:

2024-10-28T02:36:17.498275Z ERROR tabby_webserver::service::answer: ee/tabby-webserver/src/service/answer.rs:167: Failed to get chat completion chunk: JSONDeserialize(Error("unknown variant ``, expected one of `stop`, `length`, `tool_calls`, `content_filter`, `function_call`", line: 1, column: 217))

[wapleeeeee]

The llama.cpp server worked. But for more efficient in our system, Do you have plan to:

1. Update llama.cpp version to support Minicpm3-4B.
2. Make OpenAI completion kind work via vllm.
3. Support sglang for models http api.

Any progress in these aspects will be highly beneficial to us. Looking forward to your prompt reply. Thank you.

[zwpaper]

Thank you for exploring the various backends for testing with Tabby.

Regarding your questions:

1. Updating llama.cpp is on our agenda and will happen soon.
2. The vllm integration has been completed, and we hope to release it with version 0.19.0.
3. I'm not very familiar with sglang, but I conducted some quick research and found notes indicating that it has an experimental OpenAI-compatible API. Perhaps we can try it out later.

Feel free to ask if you need any further modifications!