Wrong translation of the Helsinki-NLP/opus-mt-es-en encoder and decoder in ONNX

[Zapotecatl]

System Info

• transformers version: 4.28.1
• Platform: Windows-10-10.0.19045-SP0
• Python version: 3.9.7
• Huggingface_hub version: 0.14.1
• Safetensors version: not installed
• PyTorch version (GPU?): 2.0.1+cu117 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help?

@ArthurZucker

I have exported the model Helsinki-NLP/opus-mt-es-en in onnx to translate spanish to english with this command:
optimum-cli export onnx --model Helsinki-NLP/opus-mt-es-en D:\\Marian

With that command I get the encoder and decoder. However, I get a wrong translate in my python program, I was wondering if you can help me to adress my problem.

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Export command:

optimum-cli export onnx --model Helsinki-NLP/opus-mt-es-en D:\\Marian

Program:

import onnxruntime as rt
import numpy as np
from transformers import AutoTokenizer

session_encoder = rt.InferenceSession('D:\\Marian\\encoder_model.onnx')
session_decoder = rt.InferenceSession('D:\\Marian\\decoder_model.onnx')

tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-es-en")
encoded_input = tokenizer("Crimen y castigo está dividida en seis partes más el epílogo. Mucho se ha comentado de la noción de dualismo en la obra, sugiriéndose la existencia de cierto grado de simetría en ella. Los episodios clave se distribuyen primero en una mitad y luego de nuevo en la otra.",return_tensors="np", padding=True)

input_ids = np.array(encoded_input.input_ids).astype(np.int64).reshape(1, -1)
attention_mask = np.array(encoded_input.attention_mask).astype(np.int64).reshape(1, -1)

encoder_input = {
    'input_ids': input_ids,
    'attention_mask': attention_mask
}

last_hidden_state = session_encoder.run(None, encoder_input)[0]

size = 60
decoder_input_ids = np.full((1, size), 65000).astype(np.int64)

decoder_input = {
    'encoder_attention_mask': attention_mask,
    'input_ids': decoder_input_ids,
    'encoder_hidden_states': last_hidden_state
}

for i in range(1, size):
    logits = session_decoder.run(None, decoder_input)[0]
    tokens = logits.argmax(axis=2)[0] # greedy_search
    decoder_input["input_ids"][0, i] = tokens[i]

decoded_output = tokenizer.decode(decoder_input["input_ids"].reshape(-1), skip_special_tokens=True)
print(decoded_output)

Expected behavior

The output:
The novel Crime and Punishment is divided into six parts plus the epilogue. Much has been commented on the notion of dualism in the work, suggesting the existence of a certain degree of symmetry in it. Key episodes are first distributed in one half and then again in the other.

However, the current output is:
Crime Crime Punishment is divided six parts more epi.... of of,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,

[ArthurZucker]

Hey! Thanks for reporting, this is a duplicate of #26271 and #26216 😉 I'll try to adresse this asap

[ArthurZucker]

I opened PRs for the models that were affected online, mostly big and base models. Now the conversion script just needs to be updated to reflect this change!

[ArthurZucker]

I don't consider it done yet since I since the script needs a bit of rework. I'll share the script I used to get the correct checkpoints and automatically update them gist

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.