gguf-py: Add support for loading merges.txt

[KerfuffleV2]

I added support in gguf.py for loading merges.txt when loading merges is enabled and merges aren't found in tokenizer.json. It will also print out a warning when loading merges is turned on but there no merges are found (to hopefully avoid accidentally creating a model that can't be loaded).

I also added a --padvocab option to convert.py which will just create <dummy00001> etc tokens when the model expects more vocab entries than exist in the vocab metadata.

Some other minor cleanups. (I don't know about the line splitting format but some of the lines were getting absurdly long.) Also fixed a small bug where the endianness option wasn't honored when creating vocab only GGUF files.

edit: Confirmed to fix CausalLM models. Tested on https://huggingface.co/CausalLM/EarlyFailures14B/tree/main creating a vocab-only model, which presumably should be the same as the real CausalLM models.

gguf: WARNING: merges.txt: Line 109171: Entry malformed, ignoring
Padding vocab with 213 token(s) - <dummy00001> through <dummy00213>
This gguf file is for Little Endian only
gguf: Adding 109170 merge(s).
gguf: Setting special token type bos to 151643
gguf: Setting special token type eos to 151643
gguf: Setting special token type pad to 151643

Tried loading:

llm_load_vocab: mismatch in special tokens definition ( 421/152064 vs 213/152064 ).
llm_load_print_meta: format           = unknown
llm_load_print_meta: arch             = llama
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 109170

I don't know how much of an issue the special tokens mismatch is. I can't see if it's the same in the master version or not since the model can't be loaded.

Closes #3732.

[CausalLM]

Yes, it works well. Thank you for your work!

[KerfuffleV2]

Yes, it works well.

Thanks for testing! Do you mean you were able to convert the model and get good results actually running it as well?

[CausalLM]

Yes，it works well on chatml format, with Q4_0 model of CausalLM/14B.

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

<|endoftext|>

<|im_start|>system 
You are a helpful assistant. 
<|im_end|> 
<|im_start|>user 
Who are you? 
<|im_end|> 
<|im_start|>assistant
Hello, I am an artificial intelligence and a large language model, also known as a conversatioto provide helpful information and answer questions to the best of my ability.

I can answer a variety of questions on various topics, including but not limited to:

- General knowledge
- Science and technology
- History and culture
- Entertainment and arts
- Health and lifestyle
- Business and finance
- Education and career
- Travel and leisure

I can also generate creative content such as poems, stories, scripts, code, music, and more.

Please feel free to ask me any questions or request my assistance. I will do my best to provide you with accurate and comprehensive information.<|endoftext|>


llama_print_timings:        load time =    1728.12 ms
llama_print_timings:      sample time =     147.91 ms /   152 runs   (    0.97 ms per token,  1027.65 tokens per second)
llama_print_timings: prompt eval time =    1263.65 ms /    50 tokens (   25.27 ms per token,    39.57 tokens per second)
llama_print_timings:        eval time =   38164.05 ms /   152 runs   (  251.08 ms per token,     3.98 tokens per second)
llama_print_timings:       total time =  132450.43 ms

The special tokens <|endoftext|> <|im_start|> and so on, all work as expected, also CJK (Chinese) characters.

[KerfuffleV2]

Thanks! One thing looks a bit weird though:

"also known as a conversatioto provide helpful information" - Did the model actually generate that, or it just not pasted correctly?

[CausalLM]

system_info: n_threads = 6 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | 
main: interactive mode on.
sampling: 
        repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
        top_k = 40, tfs_z = 1.000, top_p = 0.950, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = 2560, n_keep = 0


== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to LLaMa.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

<|endoftext|>

<|im_start|>system 
You are a helpful assistant. 
<|im_end|> 
<|im_start|>user 
Who are you? 
<|im_end|> 
<|im_start|>assistant
Hello, I am an AI language model created to assist users like you with various tasks and provide information. I can help answer questions, offer suggestions, or engage in conversations on a wide range of topics. If you have any specific needs or desires for discussion, please feel free to let me know, and I'll do my best to accommodate you.<|endoftext|>

Cannot reproduce that, maybe it is just an accident. But it is quite a large ver Q4_0, the typo error should not happen.

[TheBloke]

Quick question: will I be able to pass --padvocab unconditionally, and rely on it to pad vocab if necessary? Or do I need to check if it's required first?

(I'd love it if I can just set it always, that'd be really helpful and allow me to remove a bunch of complicated code that checks if the vocab size is correct and has been really unreliable for me as models keep coming out with custom vocab sizes)

[KerfuffleV2]

@TheBloke

will I be able to pass --padvocab unconditionally, and rely on it to pad vocab if necessary?

Yes, it'll only activate in the case the "vocab size mismatch" error would have triggered previously. So as long as padding is always what you want to do, you can just unconditionally specify it.

However, there aren't really any checks. Just for example, suppose no vocab could be loaded from the metadata (but it didn't cause an actual error, maybe the JSON keys were just empty or whatever) and the model expects 32,000 vocab entries, well... You'll successfully generate a model with a vocab of <dummy00001> through <dummy32000> which probably won't work so great. :)

[KerfuffleV2]

Cannot reproduce that, maybe it is just an accident.

If you still have the output of the first run in your console, you can scroll back near the top and find the seed that was used to generate it. It will look like:

Log start
main: build = 1420 (13e08d0)
main: built with cc (GCC) 13.2.1 20230801 for x86_64-pc-linux-gnu
main: seed  = 1698064864

So if you want to try to reproduce it, you can try using --seed 1698064864 or whatever.

[cebtenzzre]

If you still have the output of the first run in your console,

This information is also logged to a file that looks like main.139870073946112.log.

[KerfuffleV2]

Is this relevant for this discussion?

Hmm, hard to say. You didn't provide information about the prompt, generation seed, sampling settings, etc so there isn't really a way to reproduce what you did. Also it seems like you're using llama.cpp through a Python wrapper. It might be built with an older version. I know there were a number of BPE vocab/tokenizer fixes fairly recently.

Please try to reproduce your issue using llama.cpp directly (like the main example) and the latest release.

[goerch]

I know there were a number of BPE vocab/tokenizer fixes fairly recently.

We are still working actively on it.

[TheBloke]

@JeremyBickel

I've not seen that in my testing of the new GGUFs. But it could depend what kind of characters are output I guess. And yes, please show the exact prompt and parameters used.

Here's an example of a command I know works at my end:

./main -m /workspace/process/causallm_14b/gguf/causallm_14b.Q5_1.gguf -p "<|im_start|>system\nA conversation between a user and an LLM-based AI assistant. The assistant gives helpful and honest answers.<|im_end|>\n<|im_start|>user\nWhat is The HitchHikers Guide To The Galaxy?<|im_end|>\n<|im_start|>assistant"

Output:

<|im_start|>system\nA conversation between a user and an LLM-based AI assistant. The assistant gives helpful and honest answers.<|im_end|>\n<|im_start|>user\nWhat is The HitchHikers Guide To The Galaxy?<|im_end|>\n<|im_start|>assistant:The Hitchhiker's Guide to the Galaxy is a science fiction comedy series written by Douglas Adams, consisting of five novels published between 1978 and 1982. The story follows the adventures of Arthur Dent, a man who has lost most of his life after the Earth was destroyed to make way for a hyperspace bypass. He is saved by Ford Prefect, an alien researcher from the fictional publishing house known as "The Hitchhiker's Guide to the Galaxy," and they embark on a journey through space with other characters such as Zaphod Beeblebrox, the President of the Galaxy, Trillian, a human woman with connections to various alien races, and Marvin the Paranoid Android. Along the way, they encounter various planets, aliens, and challenges, all while providing humorous commentary on humanity, life, and the universe. The books are also accompanied by a radio drama, a television series, and a movie adaptation.<|endoftext|> [end of text]

(There's a few hallucinations in that output. We're noticing quite a lot of hallucination with CausalLM 7B and 14B, unfortunately)

[goerch]

ERROR: byte not found in vocab:

Time for a tokenizer test?

[TheBloke]

@JeremyBickel is that ctransformers? It's not been updated since September 10th so I wouldn't be sure it's still guaranteed to work with latest GGUFs. I'd say you should try llama-cpp-python, but I've seen someone reporting that same bug with that as well.

Maybe only llama.cpp supports this properly right now.

Running your prompt on the latest llama.cpp gave:

<|im_start|>system
You are a helpful assistant.
<|im_end|>
<|im_start|>user
Tag each word of the following SENTENCE using these semantic tags: agent, main-action, purpose, direct-object, theme, relative pronon, status-action, location, recipient, comparison marker, referential-agent, transitive-action, means marker, time-indicator, description, status, conjunction, theme. SENTENCE: And after these things I saw four angels standing on the four corners of the earth, holding the four winds of the earth, that the wind should not blow on the earth, nor on the sea, nor on any tree.
<|im_end|>
<|im_start|>assistant:And (conj) after (time-indicator) these (theme) things (theme) I (referential-agent) saw (status-action) four (numeral) angels (theme) standing (status-action) on (location) the (theme) four (numeral) corners (theme) of (preposition) the (theme) earth (theme), holding (transitive-action) the (theme) four (numeral) winds (theme) of (preposition) the (theme) earth (theme), that (relative pronon) the (theme) wind (theme) should (status-action) not (modal) blow (direct-object) on (preposition) the (theme) earth (theme), nor (conjunction) on (preposition) the (theme) sea (theme), nor (conjunction) on (preposition) any (numeral) tree (theme).<|endoftext|> [end of text]

[JeremyBickel]

@TheBloke
I confirm that. I got the binary llama.cpp and ran main. It worked. So, maybe it it ctransformers. Sorry for the gigantic time sink. ..and thanks.

[goerch]

From CausalML 's special_tokens_map :

  "additional_special_tokens": [
    "<|im_start|>",
    "<|im_end|>"
  ],

But I see them in output of main : is this expected behavior?

[KerfuffleV2]

But I see them in output of main : is this expected behavior?

Unfortunately, I can't really answer questions like that since all I did was add support for loading merges from a different source.

I guess the thing to do would be to run it with the official Python stuff and compare how tokenizing occurs with what llama.cpp does.

[goerch]

I guess the thing to do would be to run it with the official Python stuff and compare how tokenizing occurs with what llama.cpp does.

I'm on it :)

[KerfuffleV2]

Although adding stuff like tests for specific models is good and certainly important, any further changes needed with this pull specifically that just adds support for loading merges from merges.txt?

[TheBloke]

Although adding stuff like tests for specific models is good and certainly important, any further changes needed with this pull specifically that just adds support for loading merges from merges.txt?

I for one would love to see this merged, for --padvocab and any future merges.txt based models.

[KerfuffleV2]

@TheBloke

I for one would love to see this merged,

I too yearn for closure! Have you been using this? In other words, can I assume that everything still works fine with other changes that have occurred during this time. I know it merges cleanly but that's not always a guarantee.

[sorgfresser]

I can verify this fixes #3935 for me and while using @KerfuffleV2 's fork is an option for now, I would really like to see this merged as I am certainly not the only one trying to convert models with a non-matching vocab size and converting, then switching back to the original repo is no real solution.

[TheBloke]

Sorry @KerfuffleV2 I haven't actually tried it yet :) I was kind of waiting for it to be merged, as I still have my own workaround code in place

I have however used the implementation of --padvocab that strutive07 provided as part of their HF Vocab module, and can confirm that that worked great. I know it's not the same code of course..

I will do a manual test of this PR shortly and let you know.

[KerfuffleV2]

No problem. I don't really know which pull request is going to actually get merged (first) or whatever. We won't want two implementations of the vocab padding stuff, so if the other one gets merged first then I'll want to remove it from this pull and likely the same for the other one as well.

---

edit:

I know it's not the same code of course..

The padding stuff is same code, so we can say that part was tested sufficiently.

The merges.txt part is just a fallback and only applies if the other source of merges doesn't provide any. So there's not too much chance of that causing issues.

More testing is, of course, always welcome!

[TheBloke]

OK I've done some testing and --padvocab is working well on all the models I've thrown at it. And the BPE merges seems fine as well. Great!

Only issue I noticed is a tangential one - the problem of added_tokens with an ID that's less than vocab_size. This was meant to be resolved by another PR, but I don't know what happened to that; I've lost track of which PR it was in, and I don't think it's been merged

So currently models like Open-Orca/Mistral-7B-OpenOrca will fail with:

 [pytorch2] tomj@MC:/workspace/git/kerfuffle-llamacpp (feat-use-mergestxt ✔) ᐅ ./convert.py /workspace/models/Open-Orca_Mistral-7B-OpenOrca --outtype f16 --outfile /workspace/models/oo-mistral.fp16.gguf --padvocab
Loading model file /workspace/models/Open-Orca_Mistral-7B-OpenOrca/pytorch_model-00001-of-00002.bin
Loading model file /workspace/models/Open-Orca_Mistral-7B-OpenOrca/pytorch_model-00001-of-00002.bin
Loading model file /workspace/models/Open-Orca_Mistral-7B-OpenOrca/pytorch_model-00002-of-00002.bin
params = Params(n_vocab=32002, n_embd=4096, n_layer=32, n_ctx=32768, n_ff=14336, n_head=32, n_head_kv=8, f_norm_eps=1e-05, f_rope_freq_base=10000.0, f_rope_scale=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('/workspace/models/Open-Orca_Mistral-7B-OpenOrca'))
Loading vocab file '/workspace/models/Open-Orca_Mistral-7B-OpenOrca/tokenizer.model', type 'spm'
Traceback (most recent call last):
  File "/workspace/git/kerfuffle-llamacpp/./convert.py", line 1224, in <module>
    main()
  File "/workspace/git/kerfuffle-llamacpp/./convert.py", line 1203, in main
    vocab = load_vocab(vocab_dir, args.vocabtype)
  File "/workspace/git/kerfuffle-llamacpp/./convert.py", line 1105, in load_vocab
    return SentencePieceVocab(path, added_tokens_path if added_tokens_path.exists() else None)
  File "/workspace/git/kerfuffle-llamacpp/./convert.py", line 372, in __init__
    raise Exception(f"Expected added token IDs to be sequential and start at {vocab_size}; got {actual_ids}")
Exception: Expected added token IDs to be sequential and start at 32000; got [0, 1, 2, 32000, 32001]

I do have code to handle that but it would be great to have that sorted at the same time as all the other convert.py changes.

Although I'm a little confused about what the longer term plan is for convert.py and vocab. Is it expected that all three of SPM, BPE and HF will be supported separately? Or that HF AutoTokenizer will supersede the other two? Because strutive's PR seems to always use AutoTokenizer, and SentencePiece isn't used at all any more (if I'm understanding correctly). And it therefore doesn't have this issue.

So if the plan is to use HF then issues like the SPM loader not handling tokens < vocab_size won't be an issue. But if it's planned to support all three as separate options, it is.

I guess I'll raise that on the HF Vocab PR as well.

[KerfuffleV2]

@TheBloke It actually did get merged: #3831

The person that made it kind of disappeared so I made a version that could apply to master and GG closed the original pull and merged those changes.

Your problem with this pull at least is that while it merged cleanly into master it hadn't been rebased on the current version in a while. I just did that. Sorry, didn't really think about it until now.

I don't have control over #3633 but you can ask to have it rebased also (or maybe it'll get merged soon?)

edit: I can't really say anything about that. My interaction with it has just been to add some features, fix some things. Unfortunately there's no long-term plan from me. (Obviously it would be good if one existed though.)

edit: Adding another random thing since I'm already hitting you with a notification. About the Yi model, apparently it gets pretty confused when it sees a BOS token and our conversion doesn't support the add_bos_token/add_eos_token options. If you want to mess with it manually, what would probably work is just setting the BOS token to 2 (same as EOS). They said they didn't use BOS at all in their training.

[KerfuffleV2]

The merges.txt stuff is in #3981, theoretically the --padvocab option will get added when #3633 is merged. This pull is now redundant.

[MiyazonoKaori]

hi, @KerfuffleV2 @CausalLM @TheBloke
I recently converted the Qwen model into a llama structure.
However, I encountered an issue with additional bias parameters in the attention KQVO within the model, which are not directly compatible with llama.cpp.
I have made the following modifications to the llama.cpp file

struct llama_layer {
    ...
    // attention bias
    struct ggml_tensor * bo;
    struct ggml_tensor * bqkv;

    //add
    struct ggml_tensor * wq_bias;
    struct ggml_tensor * wk_bias;
    struct ggml_tensor * wv_bias;
    struct ggml_tensor * wo_bias;
...
...
case LLM_ARCH_LLAMA:
case LLM_ARCH_REFACT:
    ...
    layer.wq = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_Q,   "weight", i), {n_embd, n_embd},     backend_split);
    layer.wk = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_K,   "weight", i), {n_embd, n_embd_gqa}, backend_split);
    layer.wv = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_V,   "weight", i), {n_embd, n_embd_gqa}, backend_split);
    layer.wo = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_OUT, "weight", i), {n_embd, n_embd},     backend_split);

    // add
    layer.wq_bias = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_Q,   "bias", i), {n_embd}, backend);
    layer.wk_bias = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_K,   "bias", i), {n_embd_gqa}, backend);
    layer.wv_bias = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_V,   "bias", i), {n_embd_gqa}, backend);
    layer.wo_bias = ml.create_tensor(ctx, tn(LLM_TENSOR_ATTN_OUT, "bias", i), {n_embd}, backend);
...
struct ggml_cgraph * build_llama()
...
// self-attention
{
     //add
     struct ggml_tensor * Bq = model.layers[il].wq_bias;
     struct ggml_tensor * Bk = model.layers[il].wk_bias;
     struct ggml_tensor * Bv = model.layers[il].wv_bias;

     // compute Q and K and RoPE them
     struct ggml_tensor * Qcur = ggml_mul_mat(ctx0, model.layers[il].wq, cur);
     Qcur = ggml_add(ctx0, Qcur, Bq);  // add
     cb(Qcur, "Qcur", il);

     struct ggml_tensor * Kcur = ggml_mul_mat(ctx0, model.layers[il].wk, cur);
     Kcur = ggml_add(ctx0, Kcur, Bk);  //add
     cb(Kcur, "Kcur", il);

     struct ggml_tensor * Vcur = ggml_mul_mat(ctx0, model.layers[il].wv, cur);
     Vcur = ggml_add(ctx0, Vcur, Bv);  //add
     cb(Vcur, "Vcur", il);

     Qcur = ggml_rope_custom(
          ctx0, ggml_reshape_3d(ctx0, Qcur, n_embd_head, n_head,    n_tokens), inp_pos,
          n_embd_head, 0, 0, n_orig_ctx, freq_base, freq_scale,
          ext_factor, attn_factor, beta_fast, beta_slow
     );
     cb(Qcur, "Qcur", il);

     Kcur = ggml_rope_custom(
           ctx0, ggml_reshape_3d(ctx0, Kcur, n_embd_head, n_head_kv, n_tokens), inp_pos,
           n_embd_head, 0, 0, n_orig_ctx, freq_base, freq_scale,
           ext_factor, attn_factor, beta_fast, beta_slow
     );
     cb(Kcur, "Kcur", il);
     llm_build_kv_store(ctx0, hparams, kv_self, gf, Kcur, Vcur, n_ctx, n_tokens, kv_head, cb, il);
     cur = llm_build_kqv(ctx0, hparams, kv_self,
                                     model.layers[il].wo, 
                                     model.layers[il].wo_bias,  //add
     Qcur, KQ_scale, KQ_mask, n_ctx, n_tokens, n_kv, -1.0f, cb, il);
     cb(cur, "kqv_out", il);

After these changes, I was able to successfully generate with the llama.cpp.
Unfortunately, the output from the model is highly unsatisfactory and appears completely disordered.

sampling: 
	repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
	top_k = 40, tfs_z = 1.000, top_p = 0.700, min_p = 0.050, typical_p = 1.000, temp = 0.800
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = 2000, n_keep = 0


<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nWho are you?\n<|im_end|>\n<|im_start|>assistant
I
Who am<|im_end|> [end of text]

Could anyone kindly advise where I might have gone wrong? What modifications should I consider to enable the model to function correctly?

Thank you very much for your time and assistance.