Idefics 2 fixed bug shown up again

[hukkai]

System Info

• transformers version: 4.45.1
• Platform: Linux-5.4.0-139-generic-x86_64-with-glibc2.31
• Python version: 3.10.14
• Huggingface_hub version: 0.23.4
• Safetensors version: 0.4.3
• Accelerate version: 0.31.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.2.2+cu121 (True)
• Tensorflow version (GPU?): not installed
• Flax version (CPU?/GPU?/TPU?): not installed
• Jax version: not installed
• JaxLib version: not installed
• Using distributed or parallel set-up in script?: No
• Using GPU in script?: No
• GPU type: NVIDIA A100 80GB PCIe

Who can help?

@amyeroberts @qubvel

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Just run the official code from https://huggingface.co/HuggingFaceM4/idefics2-8b for idefics2-8b can reproduce the bug.

Expected behavior

File ~/miniconda3/envs/llava/lib/python3.10/site-packages/transformers/models/idefics2/modeling_idefics2.py:1296, in Idefics2Model.inputs_merger(self, input_ids, inputs_embeds, image_hidden_states)
1294 new_inputs_embeds = inputs_embeds.clone()
1295 reshaped_image_hidden_states = image_hidden_states.view(-1, vision_hidden_size)
-> 1296 new_inputs_embeds[special_image_token_mask] = reshaped_image_hidden_states
1297 return new_inputs_embeds

RuntimeError: shape mismatch: value tensor of shape [640, 4096] cannot be broadcast to indexing result of shape [0, 4096]

The bug was first reported in #31380. I can confirm it is fixed in transformers version: 4.45.0-dev. However, as I upgraded to 4.45.1, it shows up again.

[amyeroberts]

Hi @hukkai, thanks for reporting!

I've opened #33776 which should resolve this issue. For reference, same issue reported in #33752

[aroun-coumar]

Hey @amyeroberts I have created a PR before you created and got a review approved
Is this conflict okay?

Thanks

[aroun-coumar]

#33766 This is the PR
May i merge this

[amyeroberts]

Hi @aroun-coumar, apologies for the duplicated work. I saw #33752 first, and so opened a PR for that issue.

Your PR was opened first, so let's merge that one in.