[Whisper Tokenizer] Skip special tokens when decoding with timestamps

[sanchit-gandhi]

What does this PR do?

Decoding with timestamps always returns the special tokeniser tokens, ignoring the argument skip_special_tokens. This PR fixes this behaviour to respect skip_special_tokens, both with and without timestamp decoding.

Before:

from transformers import WhisperTokenizer

tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-tiny")

encoded_input = [
    50258, 50363, 50364, 634, 575, 12525, 22618, 1968, 6144,
    35617, 20084, 1756, 311, 589, 307, 534, 10281, 934,
    439, 293, 50676, 50676, 393, 4411, 294, 309, 457,
    707, 295, 33301, 286, 392, 6628, 13, 50836, 50257,
]

tokenizer.decode(encoded_input, decode_with_timestamps=True, skip_special_tokens=True)

Output:

<|startoftranscript|><|notimestamps|><|0.00|> He has grave doubts whether Sir Frederick Layton's work is really Greek after all and<|6.24|><|6.24|> can discover in it but little of rocky Ithaca.<|9.44|><|endoftext|>

Now:

tokenizer.decode(encoded_input, decode_with_timestamps=True, skip_special_tokens=True)

Output:

<|0.00|> He has grave doubts whether Sir Frederick Layton's work is really Greek after all and<|6.24|><|6.24|> can discover in it but little of rocky Ithaca.<|9.44|>

[ArthurZucker]

Nice catch and thanks for adding tests 😉

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

Thanks for adding this!