Fix pix2struct

[IlyasMoutawwakil]

What does this PR do?

In #34089 some stuff might have not being copied correctly from T5. In pix2struct case past_key_value was being overridden by the first attention block, resulting in failure in cross attention.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[echarlaix]

In transformers v4.46 generation for Pix2Struct fail when setting use_cache=True

Code to reproduce this issue : https://huggingface.co/docs/transformers/v4.46.0/en/model_doc/pix2struct#transformers.Pix2StructForConditionalGeneration.forward.example with use_cache=True

from PIL import Image
import requests
from transformers import AutoProcessor, Pix2StructForConditionalGeneration
model_id = "google/pix2struct-textcaps-base"

processor = AutoProcessor.from_pretrained(model_id)
model = Pix2StructForConditionalGeneration.from_pretrained(model_id)
url = "https://www.ilankelman.org/stopsigns/australia.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(images=image, return_tensors="pt")
generated_ids = model.generate(**inputs, max_new_tokens=50, use_cache=True)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Thanks for fixing! Very bad that we don't have generation test mixin in Pix2Struct 😢 , will add tests in the next batch with some left-over VLMs

[ArthurZucker]

Thanks! can you add the snippet as a test? 🤗

[IlyasMoutawwakil]

@ArthurZucker I added a small quick test

[IlyasMoutawwakil]

there's one thing I don't understand, running the same test with T5 passes even without eval mode, but it was necessary to get it passing for pix2struct.

[ArthurZucker]

Thanks, did not have the chance to approve but LGTM 🤗