Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Orion-14B chat template is not support #6009

Closed
power0341 opened this issue Mar 12, 2024 · 0 comments · Fixed by #6066
Closed

Orion-14B chat template is not support #6009

power0341 opened this issue Mar 12, 2024 · 0 comments · Fixed by #6066
Assignees

Comments

@power0341
Copy link

The chat template for Orion's models is missing, and applying chatml format will give wrong response.

./build/bin/server -m ./Orion-14B-Chat.gguf -c 2048

image

curl --request POST --url http://localhost:8080/completion --header "Content-Type: application/json" --data '{"prompt": "
Write a c++ program printing G'day.","n_predict": 512}'

output:

{"content":"\nHere is a simple C++ program that prints \"Hello, World!\": \n\n```c++ \n#include <iostream> \n#include <iostream> Hello, this code snippet.\n#include hello world program to print \"Hello, using c++ Hello!","generation_settings":{"dynatemp_exponent":1.0,"dynatemp_range":0.0,"frequency_penalty":0.0,"grammar":"","ignore_eos":false,"logit_bias":[],"min_keep":0,"min_p":0.05000000074505806,"mirostat":0,"mirostat_eta":0.10000000149011612,"mirostat_tau":5.0,"model":"../ollama_wks/Orion-14B-Chat-Q2_K.gguf","n_ctx":2048,"n_keep":0,"n_predict":-1,"n_probs":0,"penalize_nl":true,"penalty_prompt_tokens":[],"presence_penalty":0.0,"repeat_last_n":64,"repeat_penalty":1.100000023841858,"samplers":["top_k","tfs_z","typical_p","top_p","min_p","temperature"],"seed":4294967295,"stop":[],"stream":false,"temperature":0.800000011920929,"tfs_z":1.0,"top_k":40,"top_p":0.949999988079071,"typical_p":1.0,"use_penalty_prompt_tokens":false},"id_slot":0,"model":"../ollama_wks/Orion-14B-Chat-Q2_K.gguf","prompt":"Write a c++ program printing hello world.","stop":true,"stopped_eos":true,"stopped_limit":false,"stopped_word":false,"stopping_word":"","timings":{"predicted_ms":25227.774,"predicted_n":69,"predicted_per_second":2.7350807883406594,"predicted_per_token_ms":365.6199130434783,"prompt_ms":1169.981,"prompt_n":9,"prompt_per_second":7.692432612153531,"prompt_per_token_ms":129.9978888888889},"tokens_cached":77,"tokens_evaluated":9,"tokens_predicted":69,"truncated":false}

gg model and chat template can be found here.

"{% for message in messages %}{% if loop.first %}{{ bos_token }}{% endif %}{% if message['role'] == 'user' %}{{ 'Human: ' + message['content'] + '\n\nAssistant: ' + eos_token }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token }}{% endif %}{% endfor %}"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants