Mixtral manual head_dim

[wavy-jung]

Feature request

transformers/src/transformers/models/mixtral/modeling_mixtral.py

Line 284 in 816f442

self.head_dim = self.hidden_size // self.num_heads

head_dim in mixtral model is forced to have the value of hidden_size // num_heads. However, this it not the case in llama model or even in mistral model. So, it will be a good minor feature to support manual head_dim setting for mixtral model as well!

Motivation

• manual head_dim in llama or mistral model

Your contribution

PR