Can't convert-checkpoint Mistral 7B v0.3: safetensors_rust.SafetensorError: File does not contain tensor model.embed_tokens.weight

[Ace-RR]

System Info

on H100 Nvidia

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

1.git -C /workspace clone https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3

2.python tensorrt_llm/examples/llama/convert_checkpoint.py --model_dir /workspace/Mistral-7B-Instruct-v0.3 --output_dir /workspace/trt_ckpt/mistral3/fp16 --dtype bfloat16

Expected behavior

0.11.0.dev2024060400
Total time of converting checkpoints: xx:xx:xx

actual behavior

[TensorRT-LLM] TensorRT-LLM version: 0.11.0.dev2024060400
0.11.0.dev2024060400
Traceback (most recent call last):
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 439, in
main()
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 431, in main
convert_and_save_hf(args)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 366, in convert_and_save_hf
execute(args.workers, [convert_and_save_rank] * world_size, args)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 390, in execute
f(args, rank)
File "/app/tensorrt_llm/examples/llama/convert_checkpoint.py", line 355, in convert_and_save_rank
llama = LLaMAForCausalLM.from_hugging_face(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/model.py", line 292, in from_hugging_face
weights = load_weights_from_hf_safetensors(hf_model_dir, config)
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1577, in load_weights_from_hf_safetensors
weights['transformer.vocab_embedding.weight'] = load(
File "/usr/local/lib/python3.10/dist-packages/tensorrt_llm/models/llama/convert.py", line 1555, in load
res = safetensors_ptrs[ptr_idx].get_tensor(key)
safetensors_rust.SafetensorError: File does not contain tensor model.embed_tokens.weight

additional notes

Mistral 7B v0.3 requires transformers 4.42.0.dev0

version of transformers with tensorrtllm_backend: 4.40.2

But the command doesn't work even if transformers is upgrade to 4.42.0.dev0

[hijkzzz]

@nv-guomingz This is a new feature due to changes in the weight format and tokenizer of mistral v0.3

[nv-guomingz]

hi @Ace-RR , a quick war is to delete the consolidated.safetensors from you local model directory.
Or you can apply below changes on your side.

[nv-guomingz]

Hi @Ace-RR
we've merged the fixing into internal repo and please fetch the next weekly update for your further work.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."

[nv-guomingz]

Hi @Ace-RR please feel free to reopen it if needed