Fix the incorrect permutation of gguf

[PenutChen]

What does this PR do?

Fixes #31766. The permutation of q_proj and k_proj needs to consider both num_attention_heads and num_key_value_heads.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc

[LysandreJik]

Thanks for your PR @PenutChen, we really appreciate it!

@SunMarc it'd be awesome if you can review this when you have a minute.

[SunMarc]

Thanks for the deep dive @PenutChen Really good job ! Can you add a few tests since it was not working with a bigger llama ! If possible, we can test this on a q4 model to not take too much memory !

[PenutChen]

@SunMarc Thanks for the advice! I noticed that there is already a Q4 Llama3 test. This update will change the expected text. Should I just update the expected text?

[SunMarc]

Thanks a lot for iterating ! Could you are the perplexity of the model before and after for llama 3 ? If the llama 3 perpelxity was so high, it's a bit strange that it was able to pass the test / generate coherent text

[PenutChen]

The perplexity of Llama3 Q4 is 1713.8865 before the fix and 6.4626 after the fix. The model comes from NousResearch/Meta-Llama-3-8B-GGUF.

They still produce reasonable output if the sequence length is short. This might be because only the q_proj and k_proj weights are abnormal, and their weights are relatively low. Other weights are robust to a certain extent, which could be the cause. This is just my guess.

[SunMarc]

Thanks for the confirmation !

[LysandreJik]

Sorry for the delayed review, thanks for the PR @PenutChen!