Donut model only works on 4.36.2 for inference, not 4.37.2

[VikParuchuri]

System Info

I have a donut model with some slight customizations to the mbart decoder (added gqa and moe). It works fine on 4.36.2 and 4.37.2 for training.

But inference only works on 4.36.2. When I run inference on 4.37.2, then the output degenerates into repetition. @amyeroberts

Here is an example (the text has been ocred with donut, then rendered back onto a page image):

This is with 4.36.2:

And this is with 4.37.2:

Everything else is identical (same system, same packages). I don't see anything obvious in the release notes that would cause this.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

• Clone https://github.com/VikParuchuri/surya/tree/rec (has to be the rec branch)
• Install (see README)
• Run python benchmark/recognition.py --max 1 --debug

You should see different output with different transformers versions.

Expected behavior

I expect the output to be the same with both versions, and to not degenerate into repetition.

[ArthurZucker]

fyi @NielsRogge

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[NielsRogge]

Hi sorry for the late reply, I've checked all slow integration tests of Donut, and they all pass:

RUN_SLOW=yes pytest tests/models/vision_encoder_decoder/test_modeling_vision_encoder_decoder.py::DonutModelIntegrationTest

Which would mean that Donut works as expected on v4.38.2 dev.

This might be explained because of the customizations? Could you clarify which things are changed compared to Donut?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.