AttributeError: 'VisionAttention' object has no attribute 'head_dim'

[Betty-J]

Hello,
I am encountering the following error while fine-tuning the qwen2-vl-2b-instruct with the Swift framework:
File "./miniconda3/envs/swift/lib/python3.11/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 296, in forward
[rank1]: attn_weights = torch.matmul(q, k.transpose(1, 2)) / math.sqrt(self.head_dim)
AttributeError: 'VisionAttention' object has no attribute 'head_dim'.

After checking the original modeling_qwen2_vl.py file, I found that 'self.head_dim' is indeed not defined in VisionAttention. How can I resolve it?

[dongZheX]

when i set attn_implementation="eager", this error happened.

[simonJJJ]

@Betty-J thanks for pointing this out, we will fix it.

[simonJJJ]

we opened a PR for fixing this.

[ApolloRay]

Has it already fixed ? I meet the same question.

[nihirv]

Simple fix -- just add this line in the VisionAttention class: self.head_dim = dim // num_heads (site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py)

[lonnex]

torch版本更新到2.2.2即可