Prompt structure after the --in-prefix-bos commit

[ghost]

Hello, I'm adhereing to llama.cpp/example/main/README.txt

Here's main: ./main -m ~/wizardlm-7b-v1.0-uncensored.ggmlv3.q4_0.bin -i -r "User:" --in-prefix " " --in-suffix "Assistant:" -f ~/storage/shared/PT/Wiz.txt

Working as expected Grammar commit :

Assistant: Hi! Is there anything that you would like to discuss? User: Hi, what's the best movie?
Assistant: That is a subjective question as people have different opinions on what makes a good movie. However, some popular movies that have been widely appreciated include "The Shawshank Redemption," "The Godfather," and "Forrest Gump."
User: 

Note at the end of the Assistants message: llama.cpp inserts, User:(space), compared with --in-prefix-bos commit:

Assistant: Hi! Is there anything that you would like to discuss? User: Hi, what's the best movie?
Assistant: There are many great movies out there, but it really depends on your personal preferences and interests. Some of my favorite movies include The Shawshank Redemption, Forrest Gump, Titanic, The Godfather, and The Matrix. Is there anything specific you're looking for in a movie?
 

llama.cpp negates, "User:", and only puts a space, breaking the structure of conversation.

The directions in llama.cpp/example/main/README.txt do not work as expected since the --in-prefix-bos commit.

Thank you.

[ghost]

Same issue

[jxy]

Previously, the first reverse prompt (if existed) is always inserted. Now if the model generate EOS, the reverse prompt is not inserted. To get the same behavior, please use

./main -m "$MODEL" -i --in-prefix "User: " --in-suffix "Assistant:" ...

instead.

[ghost]

Thank you for your response! I tested last night, and it appears to be mostly working.

[jxy]

-r still works, if the model generates the reverse prompt, the code will stop the generation at that reverse prompt. It's just that the code no longer inserts the first reverse prompt if the mode generates EOS.

[ghost]

It's just that the code no longer inserts the first reverse prompt if the mode generates EOS.

Sure, I appreciate the explanation. If I understand correctly then ,-r, is obselete.

I don't know how to dicern whether or not a model generates an EOS - most model cards don't contain that information, so I'd rather not guess and use an obselete parameter.

[jxy]

-r is not obsolete.

It does exactly what it advertises to do, stopping generation when the code sees the string, which can be anything. If you work with models that do not generate EOS as needed (such as all the base models), you would need the reverse prompt.

Of course the --grammar can work like -r, but --grammar also does a lot more, while -r is simple and effective.

[ghost]

-r is not obsolete.

It does exactly what it advertises to do

Okay, so previously -r did more than advertised?

More testing shows that whether or not --in-prefix functions depends on the model. For example, how to strickly follow prompt structure for a model like Nous-Hermes-Llama-2-7B-GGML?

In this case, --in-prefix, alone doesn't allow the user to type, -r and --in-prefix produces User:User: . Readme instruction fails.

I'm torn because newer llama.cpp has memory upgrades, but old llama.cpp works as expected, and so far I don't see a way to follow prompt structure strickly with many models.

[jxy]

Please read Nous Hermes model card https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b

It's an instruction model, not a chat model.

After reading over the model card, you'll find a link to github that gives you example prompt. To strictly follow prompt structure, you unfortunately need to be able to read the Python code and reconstruct its prompt, https://github.com/teknium1/alpaca-roleplay-discordbot/blob/3595c171c2bab18feaa834518c1af990b978c49b/roleplay-bot.py#L116-L177

[ghost]

Please read Nous Hermes model card https://huggingface.co/NousResearch/Nous-Hermes-Llama2-13b

That's not the model, but I checked out the model card and it shows a Prompt Template section.

To strictly follow prompt structure, you unfortunately need to be able to read the Python code and reconstruct its prompt

There was no need reconstruct a Prompt Template previously, and now decyphering Python is required?

So @TheBloke's nous-hermes-llama-2-7b Prompt Template is incorrect?

[TheBloke]

Actually mine might be incorrect in that I don't have a newline after instruction and they do. I'll fix that.

Whatever is in the original model card will be correct, which is at the bottom of my readme. I'll update my PT section later this evening

[ghost]

Actually mine might be incorrect in that I don't have a newline after instruction and they do. I'll fix that.

@TheBloke Thanks for clarifying. I tried with an updated PT and it works.

@jxy I'll double check PT's going forward and close on this issue. Thank you.

It's notable that the, --in-prefix --in-suffix, examples from llama.cpp/example/main/README.txt are confusing.

[ghost]

Related: #2578