Llama2 models not loading (Using main branch)

[dbuos]

System Info

Error when loading the model "meta-llama/Llama-2-7b-chat-hf" using the following code:

from transformers import AutoModelForCausalLM
chk = 'meta-llama/Llama-2-7b-chat-hf'

if __name__ == '__main__':
    model = AutoModelForCausalLM.from_pretrained(chk, load_in_4bit=True, device_map='auto')
    print("Loaded Ok")

The error message was:

`do_sample` is set to `False`. However, temperature is set to 0.9 -- this flag is only used in sample-based generation modes. Set `do_sample=True` or unset temperature to continue.

This is because the method GenerationConfig.validate() raises a ValueError and that Error is not controlled in modeling_utils.py file.
One possible solution is to add the the ValueError to the except clause in that file:

Who can help?

@gante

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Using the main branch (install from source code)

from transformers import AutoModelForCausalLM
chk = 'meta-llama/Llama-2-7b-chat-hf'

if __name__ == '__main__':
    model = AutoModelForCausalLM.from_pretrained(chk, load_in_4bit=True, device_map='auto')
    print("Loaded Ok")

Expected behavior

To be able to load the model

[inconnu26]

I have same issue with another Model. I think this is urgent to fix.

[Hanzofm]

Same here. Meanwhile downgrading to transfomers 4.31 version solves the problem

[dbuos]

I found this issue in the dev branch of transformers-4.32.0.dev0. By the way, I made a PR that could solve that. #25389

[graeme204]

Same here!

[gante]

Hey everyone 👋 If you're hitting this exception, it means that there is something wrong with your model's config file 💔

Meanwhile, we are deciding internally how to massage this question into a more user-friendly solution.

[gante]

After the PR above gets merged, you will be able to do everything as before.

The only difference from before is that you will see new warnings, related to poor generate parameterization (which may come from the generation config file, as in the case of llama 2) :)