Add Qwen2Moe GGUF loading support

[VladOS95-cyber]

What does this PR do?

Add Qwen2Moe GGUF loading support

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case. Link: Community contribution: Adding GGUF support for more architectures #33260
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Regarding the task @SunMarc @LysandreJik @ArthurZucker .

[VladOS95-cyber]

Hello @SunMarc @LysandreJik @ArthurZucker! I would like to ask for a code review

[SunMarc]

Thanks for this clean PR @VladOS95-cyber ! LGTM !

[VladOS95-cyber]

Thanks for this clean PR @vanpelt ! LGTM !

Thank you for review! Why @vanpelt? =))

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[SunMarc]

Thank you for review! Why @vanpelt? =))

My bad, editted ;) usually the first name github suggests me is the author of the PR

[VladOS95-cyber]

Thank you for review! Why @vanpelt? =))

My bad, editted ;) usually the first name github suggests me is the author of the PR

No worries at all :)

[LysandreJik]

Thanks @VladOS95-cyber!

[vanpelt]

Anytime guy's 🤪. Nice work @VladOS95-cyber!

[SunMarc]

I'm hitting an error on our CI and locally for this test test_qwen2_moe_q4_0 @VladOS95-cyber . Can you have a look?
Here's the traceback:

self = <ggml.test_ggml.GgufIntegrationTests testMethod=test_qwen2_moe_q4_0>

    def test_qwen2_moe_q4_0(self):
        tokenizer = AutoTokenizer.from_pretrained(self.qwen2_moe_model_id, gguf_file=self.q4_0_qwen2_moe_model_id)
        model = AutoModelForCausalLM.from_pretrained(
            self.qwen2_moe_model_id,
            gguf_file=self.q4_0_qwen2_moe_model_id,
            device_map="auto",
            torch_dtype=torch.float16,
        )

        text = tokenizer(self.example_text, return_tensors="pt").to(torch_device)
        out = model.generate(**text, max_new_tokens=10)

        EXPECTED_TEXT = "Hello everyone, I'm a newbie here and would like"
>       self.assertEqual(tokenizer.decode(out[0], skip_special_tokens=True), EXPECTED_TEXT)
E       AssertionError: 'Hello部分齐值得关注erc区域堪称 btnCancel跳舞�ASC' != "Hello everyone, I'm a newbie here and would like"
E       - Hello部分齐值得关注erc区域堪称 btnCancel跳舞�ASC
E       + Hello everyone, I'm a newbie here and would like

tests/quantization/ggml/test_ggml.py:359: AssertionError

Also here's some interesting logs:

Some weights of Qwen2MoeForCausalLM were not initialized from the model checkpoint at RichardErkhov/Qwen_-_Qwen1.5-MoE-A2.7B-Chat-gguf and are newly initialized: ['model.layers.0.mlp.experts.0.down_proj.weight', 'model.layers.0.mlp.experts.0.gate_proj.weight', 'model.layers.0.mlp.experts.0.up_proj.weight', 'model.layers.0.mlp.experts.1.down_proj.weight', 'model.layers.0.mlp.experts.1.gate_proj.weight', 'model.layers.0.mlp.experts.1.up_proj.weight', 'model.layers.0.mlp.experts.10.down_proj.weight', 'model.layers.0.mlp.experts.10.gate_proj.weight', 'model.layers.0.mlp.experts.10.up_proj.weight', 'model.layers.0.mlp.experts.11.down_proj.weight', 'model.layers.0.mlp.experts.11.gate_proj.weight', 'model.layers.0.mlp.experts.11.up_proj.weight', 'model.layers.0.mlp.experts.12.down_proj.weight', 'model.layers.0.mlp.experts.12.gate_proj.weight', 'model.layers.0.mlp.experts.12.up_proj.weight', 
.....

Looks like we didn't manage to load the weights correctly. Also the GgufIntegrationTests.test_bloom_q8_0 test is also failing but it is easier to fix:

>       self.assertEqual(tokenizer.decode(out[0], skip_special_tokens=True), EXPECTED_TEXT)
E       AssertionError: 'Hello, I just want to say that I am just' != 'Hello, I just want to say that I am very'
E       - Hello, I just want to say that I am just
E       ?                                     ^^^^
E       + Hello, I just want to say that I am very
E       ?                                     ^^^^

tests/quantization/ggml/test_ggml.py:422: AssertionError

[VladOS95-cyber]

@SunMarc sure, I'll take a look